MySQL service periodically goes offline and gives ERROR 2002 (HY000): Can't connect to local MySQL server

Question

The Issue

I have a website running on a VPS with Ubuntu 14.04, PHP 5.5.9, and mysql Ver 12.12 Distrib 5.5.40.

About 90% of the time everything appears to be running normally with the MySQL service. Occasionally however, the MySQL service appears to go down, and it doesn't come back online on its own.

2015 Update

Since upgrading the virtual private server instance to have more RAM this issue has not presented itself again. Running Apache and a MySQL server on one box requires more than 512MB of memory to get a stable website it seems.

My Debug Steps So Far

1. I first notice the problem when I see the following error in the Apache error log:

   PHP Warning: mysqli::mysqli(): (HY000/2002): Connection refused in [...]

2. I checked the log files /var/log/mysql.err, var/log/mysql.log, and /var/log/mysql/error.log and they were empty.

3. I checked the log file /var/log/mysql/error.log.1.gz and found:

   141218 14:30:20 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 141218 14:30:20 [Note] Plugin 'FEDERATED' is disabled. 141218 14:30:20 InnoDB: The InnoDB memory heap is disabled 141218 14:30:20 InnoDB: Mutexes and rw_locks use GCC atomic builtins 141218 14:30:20 InnoDB: Compressed tables use zlib 1.2.8 141218 14:30:20 InnoDB: Using Linux native AIO 141218 14:30:20 InnoDB: Initializing buffer pool, size = 128.0M InnoDB: mmap(137363456 bytes) failed; errno 12 141218 14:30:20 InnoDB: Completed initialization of buffer pool 141218 14:30:20 InnoDB: Fatal error: cannot allocate memory for the buffer pool 141218 14:30:20 [ERROR] Plugin 'InnoDB' init function returned error. 141218 14:30:20 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 141218 14:30:20 [ERROR] Unknown/unsupported storage engine: InnoDB 141218 14:30:20 [ERROR] Aborting

4. I now check for connectivity to mysql from the command line: mysql -u root -p and get back ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)

5. Bummer. I try telnet 127.0.0.1 3306 and get telnet: Unable to connect to remote host: Connection refused

6. I also looked at netstat -a | grep mysql and get no output

7. After running sudo service mysql start everything goes back to working normally. netstat shows a running mysql service, telnet connects, mysql command line connection works, and no more PHP warnings. Site is back up to normal functionality.

8. I look at /var/log/mysql/error.log and see the following:

   141219 12:26:31 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 141219 12:26:31 [Note] Plugin 'FEDERATED' is disabled. 141219 12:26:31 InnoDB: The InnoDB memory heap is disabled 141219 12:26:31 InnoDB: Mutexes and rw_locks use GCC atomic builtins 141219 12:26:31 InnoDB: Compressed tables use zlib 1.2.8 141219 12:26:31 InnoDB: Using Linux native AIO 141219 12:26:31 InnoDB: Initializing buffer pool, size = 128.0M 141219 12:26:31 InnoDB: Completed initialization of buffer pool 141219 12:26:31 InnoDB: highest supported file format is Barracuda. InnoDB: The log sequence number in ibdata files does not match InnoDB: the log sequence number in the ib_logfiles! 141219 12:26:31 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... 141219 12:26:31 InnoDB: Waiting for the background threads to start 141219 12:26:32 InnoDB: 5.5.40 started; log sequence number 19000727 141219 12:26:32 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306 141219 12:26:32 [Note] - '127.0.0.1' resolves to '127.0.0.1'; 141219 12:26:32 [Note] Server socket created on IP: '127.0.0.1'. 141219 12:26:32 [Note] Event Scheduler: Loaded 0 events 141219 12:26:32 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.40-0ubuntu0.14.04.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 (Ubuntu)

My Desired Outcome

Not having the MySQL service go down would clearly be ideal, but at least I'd like some way of detecting if it goes down and then restarting it automatically if that happens. I've considered setting up a cron job to run a script that tries to connect to the local mysql service and then restarts the service if a connection fails. But again, it would be much better if I could just solve the root cause rather than sticking on a band-aid.

Current Theories

Right now the only clue I've found is the last 7 lines of the log file /var/log/mysql/error.log.1.gz in step 3 in my debug steps. My theory is that my VPS is overloaded at times which is causing the MySQL service to crash and burn, but I don't really want to increase my VPS size without knowing that will solve the problem. I'm hoping somebody has encountered a similar issue and could shed some light on if the service going down and not coming back is caused by lack of memory, of if that lack of memory error is a different problem.

Other StackOverflow Questions

I also looked at PHP MYSQL could not connect (HY000/2002) and mysqli::mysqli(): (HY000/2002): Can't connect to local MySQL server through socket 'MySQL' (2) but both questions seemed to only address connecting to MySQL from PHP in the first place, rather than debugging an issue with the MySQL service itself.