[support] Druapal server fatal error: "MySQL server has gone away", but DB server is still up & accessible. Help troubleshooting?

I have a Drupal 7.28 server that's not working after a number of upgrades. It's suffering from "MySQL server has gone away" errors.

This server used to work just fine. I've been unable to track down the cause of this problem, or find a fix or workaround.

Appreciate any help!

My env is

php -v PHP 5.6.0-dev (cli) (built: Jun 12 2014 10:24:30) Copyright (c) 1997-2014 The PHP Group Zend Engine v2.6.0-dev, Copyright (c) 1998-2014 Zend Technologies with Zend OPcache v7.0.4-dev, Copyright (c) 1999-2014, by Zend Technologies

mysqladmin version mysqladmin Ver 9.1 Distrib 10.0.11-MariaDB, for Linux on x86_64 Copyright (c) 2000, 2014, Oracle, SkySQL Ab and others.

Server version 10.0.11-MariaDB-log Protocol version 10 Connection Localhost via UNIX socket UNIX socket /var/cache/mariadb/mariadb.sock Uptime: 7 min 52 sec

Threads: 1 Questions: 2 Slow queries: 0 Opens: 0 Flush tables: 1 Open tables: 63 Queries per second avg: 0.004

the server's

Apache/2.4.10-dev + php-fpm

as an example of the error, from my siteroot

drush st PDOStatement::execute(): MySQL server has gone away database.inc:2171 [warning] PDOStatement::execute(): Error reading result set's header database.inc:2171 [warning] Drush command terminated abnormally due to an unrecoverable error. [error] PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away: SELECT filename FROM {registry} WHERE name = :name AND type = :type; Array ( [:name] => drush_drupal_environment [:type] => interface ) in _registry_check_code() (line 3150 of /srv/www/drupal7/dTEST/includes/bootstrap.inc).

query and web logs

==> mariadb.log <== ... 140612 12:33:07 3 Connect gk_admin@localhost as anonymous on dTEST_db 3 Query select @@version_comment limit 1 3 Query SELECT 1 3 Quit 4 Connect gk_admin@localhost as anonymous on dTEST_db 4 Query SET NAMES utf8 4 Query SET sql_mode = 'ANSI,STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER' 4 Query SELECT filename FROM registry WHERE name = 'DrupalDefaultEntityController' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'SelectQueryExtender' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'NodeController' AND type = 'class' 4 Query SELECT source FROM url_alias WHERE alias = 'node' AND language IN ('en', 'und') ORDER BY language ASC, pid DESC 4 Query SELECT alias FROM url_alias WHERE source = 'node' AND language IN ('en', 'und') ORDER BY language ASC, pid DESC 4 Query SELECT theme, pages FROM page_theme WHERE status = 1 ORDER BY weight, theme 4 Query SELECT rid, permission FROM role_permission WHERE rid IN ('1') 4 Query START TRANSACTION 4 Query SELECT filename FROM registry WHERE name = 'context_export_ui' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'ctools_export_ui' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'ctools_access_ruleset_ui' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'ctools_custom_content_ui' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'current_search_export_ui' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'panels_layouts_ui' AND type = 'class' 140612 12:33:08 4 Query SELECT filename FROM system WHERE name = 'zen' AND type = 'theme' 4 Query SELECT 1 FROM panels_layout LIMIT 0, 1 4 Query SELECT t__0.* FROM panels_layout t__0 WHERE (plugin = 'flexible') 4 Query SELECT filename FROM registry WHERE name = 'panels_mini_ui' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'services_ctools_export_ui' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'stylizer_ui' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_ui' AND type = 'class' 4 Query SELECT nt.* FROM node_type nt WHERE (disabled = '0') ORDER BY nt.type ASC 4 Query SELECT 1 FROM apachesolr_search_page LIMIT 0, 1 4 Query SELECT t__0.* FROM apachesolr_search_page t__0 4 Query SELECT * FROM apachesolr_search_page 4 Query SELECT 1 FROM services_endpoint LIMIT 0, 1 4 Query SELECT t__0.* FROM services_endpoint t__0 4 Query SELECT 1 FROM page_manager_pages LIMIT 0, 1 4 Query SELECT t__0.* FROM page_manager_pages t__0 4 Query SELECT t__0.* FROM page_manager_pages t__0 WHERE (task = 'page') 4 Query SELECT * FROM apachesolr_search_page 4 Query SELECT 1 FROM views_view LIMIT 0, 1 4 Query SELECT t__0.* FROM views_view t__0 4 Query SELECT filename FROM registry WHERE name = 'view' AND type = 'class' 4 Query SELECT * FROM views_display WHERE vid IN ('2') ORDER BY vid, position 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_display_default' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_display' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_display_page' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_display_block' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_localization_none' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_localization' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_display_feed' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_display_attachment' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_handler_field_numeric' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_handler_field' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_access_none' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_plugin_access' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_handler_argument_string' AND type = 'class' 4 Query SELECT filename FROM registry WHERE name = 'views_handler_argument' AND type = 'class' 4 Query SELECT t__0.* FROM page_manager_pages t__0 WHERE (task = 'page') 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'contact' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'search' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'file/add' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'file/%' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'block/%' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'filedepot' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'tagclouds' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'comment/%comment/devel' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'taxonomy/term/%' AND module = 'system' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'system/files' AND module = 'system' 4 Query SELECT filename FROM system WHERE name = '' AND type = 'module' 4 Query SELECT menu_name FROM menu_links WHERE router_path = 'realname/autocomplete' AND module = 'system' 4 Query DELETE FROM menu_router

==> /var/log/drupal/drupal_dTEST.log <== Jun 12 12:33:08 deskEdgar dTEST: https://dTEST.loc%7C1402601588%7Cmenu%7C127.0.0.1%7Chttps://dTEST.loc/index....: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away: DELETE FROM {menu_router}; Array#012(#012)#012 in _menu_router_save() (line 3774 of /srv/www/drupal7/dTEST/includes/menu.inc).

==> mariadb.log <== /usr/local/mariadb/bin/mysqld, Version: 10.0.11-MariaDB-log (Source distribution). started with: Tcp port: 3306 Unix socket: /var/cache/mariadb/mariadb.sock Time Id Command Argument

checking, despite the "MySQL server has gone away" error, the db server is still running

ps ax | grep mysql 11715 ? Sl 0:00 /usr/local/mariadb/bin/mysqld --defaults-file=/usr/local/etc/mariadb/my.cnf --basedir=/usr/local/mariadb --datadir=/data/db/mariadb --plugin-dir=/usr/local/mariadb/lib/plugin --log-error=/var/log/mariadb/mariadb-err.log --pid-file=/var/cache/mariadb/mariadb.pid --socket=/var/cache/mariadb/mariadb.sock --port=3306 23729 ? Ss 0:00 /bin/sh /usr/local/mariadb/bin/mysqld_safe --defaults-file=/usr/local/etc/mariadb/my.cnf

mysqladmin status Uptime: 453 Threads: 1 Questions: 1 Slow queries: 0 Opens: 0 Flush tables: 1 Open tables: 63 Queries per second avg: 0.002

Digging online, I find multiple suggestions for fixing the "MySQL server has gone away" that suggest increasing the value of `max_allowed_packet`.

I've done that,

mysqladmin variables | grep " max_allowed_packet" | max_allowed_packet | 67108864

unfortunately to no effect.

I can't get a handle yet on what the problem is here.

Any ideas or suggestions?

Grant