Our MySQL databases are backed up frequently, and I've arranged for our server to email me in the event of a failure. From our /bin/bash script which is run by crontab:

  /usr/local/mysql/bin/mysqldump -ufgw -psummat -h127.0.0.1 fgw > /home/backups/$HOUR/fgw.sql
  if [ "$?" -eq 0 ]; then
    echo "Success $HOUR fgw" > /dev/null
  else
    echo "Mysqldump encountered a problem $HOUR fgw" | mail -s "FGW BAD dump" graham@wellho.net
  fi

I got an email a couple of hours ago ... and it turned out that the MySQL server had halted. Not sure why, which frightens me, but I'm working from a cruise boat with a poor connection and the important thing was to fix it. Trying to restart:

  [root@Server51076 init.d]# ./mysql.server start
  Starting MySQL. ERROR! The server quit without updating PID file (/usr/local/mysql/data/Server51076.uk2net.com.pid).
  [root@Server51076 init.d]#

A quick trip to Google took me on to a couple of articles - which described different problems, or OS Specific things. Several referred to "Google Hell" on this subject, so let me add my own 2 cents worth in case you've arrived here from Google Hell!

It appears that what's happened is that the MySQL daemon has stopped catastrophically for some reason. It's confused into thinkig it should be referring to a PID file which didn't exist, and because it's crashed, it's refusing to restart. And it's not got any process to tidy us if I try to run server stop scripts. Thus the irritation above.

My solution to get running ...

Ensure that /etc/my.cnf contains:
  [mysqld]
  innodb_force_recovery = 1

And restart the server as this will overstep the problem above, and also recover the databases; error recovery levels of 1 or 2 are likely to let you recover (almost) everything we're told, and I have the previous good backup anyway, so I can restore if I need to.

Then stop the server once it's forced the recovery, remove the innodb_force_recovery assignment, and restart it. And - well - it worked for me. I'm monitoring carefully, and posting this to my blog partly as a test and partly in the hope it may provide a helpful clue to others in the future.
(written 2015-05-06)

 A167 - Web Application Deployment - Shell Programming (bash)
  [63] Almost like old times - (2004-09-26)
  [64] Shell Script for CGI on the web - (2004-09-26)
  [749] Cottage industry or production line data handling methods - (2006-06-07)
  [827] No news is good news with Unix and Linux - (2006-08-10)
  [1287] Work and play at Well House Manor - Football and Shell Shortcuts - (2007-08-02)
  [1345] Perl and Shell coding standards / costs of an IT project - (2007-09-11)
  [1468] Lexical v Arithemetic testing, Bash and Perl - (2007-12-11)
  [1527] Selecting file names in a shell - one word or another - (2008-02-02)
  [1904] Ruby, Perl, Linux, MySQL - some training notes - (2008-11-23)
  [3791] The Kernel, Shells and Daemons. Greek Gods in computing - (2012-07-01)
  [4400] Commenting out an echo killed my bash backup script - (2015-01-19)
  [4584] Bash ... some new scripts to - handling user input - (2015-11-27)
  [4586] Extending your bash shell with aliases, functions and extra commands - (2015-11-28)
  [4587] shell - bash. Writing conditional tests and statements - the options available - (2015-11-28)

A164 - Web Application Deployment - Services and Regular Jobs
  [544] Repeating tasks with crontab - (2005-12-27)
  [907] Browser -> httpd -> Tomcat -> MySQL. Restarting. - (2006-10-28)
  [1028] Linux / Unix - process priority and nice - (2007-01-10)
  [1288] Linux run states, shell special commands, and directory structures - (2007-08-03)
  [1553] Automatic startup and shutdown of Tomcat - (2008-02-24)
  [1633] Changing a screen saver from a web page (PHP, Perl, OSX) - (2008-05-06)
  [1700] FTP server on Fedora Linux - (2008-07-06)
  [1731] Apache httpd, MySQL, PHP - installation procedure - (2008-08-01)
  [1733] memcached - overview, installation, example of use in PHP - (2008-08-02)
  [1765] Dialects of English and Unix - (2008-08-21)
  [1903] daemons - what is running on my Linux server? - (2008-11-23)
  [2145] Using the internet to remotely check for power failure at home (PHP) - (2009-04-29)
  [2182] What Linux run level am I in? - (2009-05-15)
  [3011] What are .pid files? - (2010-10-23)
  [3143] On time - (2011-01-23)
  [3792] Managing daemons from a terminal session - (2012-07-01)

S153 - Sourcing, Running and Configuring MySQL
  [192] Current MySQL and PHP paths and upgrades - (2005-01-28)
  [334] Symbolic links and hard links - (2005-06-02)
  [489] Which MySQL server am I using? - (2005-11-07)
  [515] MySQL - an FAQ - (2005-12-03)
  [535] MySQL permissions and privileges - (2005-12-20)
  [591] Key facts - SQL and MySQL - (2006-02-04)
  [1095] Apache httpd , browser, MySQL and MySQL client downloads - (2007-02-28)
  [1123] mysqldump and mysqlrestore - (2007-03-30)
  [1131] MySQL - Password security (authentication protocol) - (2007-04-02)
  [1689] Some sideways thoughts on the news - (2008-06-27)
  [1771] More HowTo diagrams - MySQL, Tomcat and Java - (2008-08-24)
  [1935] Summary of MySQL installation on a Linux system - (2008-12-11)
  [2085] MySQL - licensing issues, even with using the name - (2009-03-16)
  [2209] Monitoring and Tuning your MySQL installation - (2009-05-31)
  [2426] Which version of MySQL am I running? - (2009-09-26)
  [2444] Potted MySQL installation - (2009-10-09)
  [2445] Securing MySQL on a production server - (2009-10-09)
  [2458] Cant connect to local MySQL server through socket /tmp/mysql.sock - (2009-10-17)
  [4390] Checking MySQL database backups have worked (not failed) - (2015-01-10)
  [4406] Fixing damaged MySQL tables - Error 1712 and Error 2013 - (2015-01-25)

Back to Newark to New York - the PATH suburban railway

Previous and next or Horse's mouth home

Forward to Election day, 2015

Web Server Admin - some of those things that happen, and solutions
What and where are the Azores?
Election results - what if we had a party list system?
Election day, 2015
Starting MySQL. ERROR! The server quit without updating PID file - how we fixed it.
Newark to New York - the PATH suburban railway
Misusing statistics? - the seedy side of election campaigning
TransAtlantic crossing - the starter
Moving from mysql to mysqli - simple worked example
People in New York