Our main web server system becomes ever more important to our business - and having procedures in place to restore systems in the event of failure becomes ever more critical.

What are we protecting against? Both hardware and software failures, and against intrusions and obsolescence (to the extent of being unuable) too. And these come in different flavours - a hardware failure (in the shape of a power supply blowing) was quickly fixed by moving the disc into a new chassis / onto a new processor, whereas a software failure that's done laten damage a time back and just surfaced can take considerable fixing!

We have three phases of backups

Phase 1
a) On server, our most dynamic of databases are backed up every hour, and our less changing databases every six hours
b) On server, our whole web site and database areas, and our /usr/local, are backed up twice a week
c) On server, a monthly backup takes in just about everything thatcan change

Phase 2
a) Our every-six-hour backup gets pulled, twice a day, onto another server (phyically in a different country)
b) Our twice a week backup similarly gets pulled onto that other server

Phase 3
From time to time, we grab copies of the backups to our own inhouse server and / or laptops

We have stopped short of transaction logging, mirror servers, etc; almost all of our server content is pretty static and the mos dynamic - the First Great Western Passenger Forum - is a free forum without income, and users shouldn't (and don't) expect robustness an order of magnitude better than the services they talk about. We lost a couple of posts the other week, and the server was down for an hour or two. Course descriptions / Well House web site hanges are uploaded from our developemnt systems for the most part, so if we loose anything on the main server we can just replace it.

Phases 1 and 2 are performed by regular time jobs under crontab - with phase 1 being shell scripts (see [here] for hourly, and [here] for twice weekly

The Phase 2 script are programmed in Expect - an easy solution to grabbing files over the network, mimicing the command line rather than some of the complexities of setting up keys and keychains. Yes, I know it's less secure, but then virtualy of all the data is public stuff anyway. The expect script is [here].

The ultimate question with all of these peocedures is "if the live system fails at the worst possible time, and in the wort possible way, how do we get it back, how long does that take, and what is the cost - financial, to the business, and in time. And those question should be the starting point as you work out your backup strategy.

(written 2015-01-24)

 A162 - Web Application Deployment - Backups and File System Management
  [153] Linux - where to put swap space - (2004-12-16)
  [334] Symbolic links and hard links - (2005-06-02)
  [554] What backup is adequate? - (2006-01-04)
  [593] Finding where the disc space has gone - (2006-02-06)
  [703] Copying files and preserving ownership - (2006-04-28)
  [735] Boys will be boys, saved by Ubuntu - (2006-05-27)
  [754] tar, jar, war, ear, sar files - (2006-06-10)
  [1013] Copy multiple files - confusing error message from cp - (2006-12-30)
  [1023] Finding public writeable things on your linux file system - (2007-01-06)
  [1288] Linux run states, shell special commands, and directory structures - (2007-08-03)
  [1439] Linux / Unix - layout of operating system files - (2007-11-20)
  [1648] The tourists guide to Linux - (2008-05-20)
  [1765] Dialects of English and Unix - (2008-08-21)
  [1801] Will your backups work if you have to restore them? - (2008-09-18)
  [1893] Some Linux and Unix tips - (2008-11-18)
  [2299] How much space does my directory take - Linux - (2009-07-20)
  [4056] An overpractical test of our backup strategy! - (2013-03-30)
  [4063] Backups by crossover between network centres - setting up automatic scp transfers - (2013-04-13)
  [4115] More or less back - what happened to our server the other day - (2013-06-14)
  [4390] Checking MySQL database backups have worked (not failed) - (2015-01-10)
  [4400] Commenting out an echo killed my bash backup script - (2015-01-19)
  [4481] Extracting data from backups to restore selected rows from MySQL tables - (2015-05-01)

T211 - Tcl/Tk - What is Expect? Why use it?
  [286] Automating regular manual procedures - (2005-04-21)
  [435] Expect for Windows - (2005-09-04)
  [1173] Cheat Sheet / Check list for Expect maintainers - (2007-05-02)
  [1174] Installing Tcl and Expect on Solaris 10 - a checklist - (2007-05-02)
  [1409] What is Expect? - (2007-10-26)
  [1411] Buffering of inputs to expect, and match order - (2007-10-27)
  [1469] Curley brackets v double quotes - Tcl, Tk, Expect - (2007-12-12)
  [1531] Expecting a item from a list of possibles - (2008-02-04)
  [1602] Automating processes through Expect - (2008-04-05)
  [2474] Using Tcl and Expect to automate repetitive jobs - (2009-10-24)
  [2489] Parallel Pinging, using Python Threads or Expect spawn lists - (2009-11-02)
  [3009] Expect in Perl - a short explanation and a practical example - (2010-10-22)
  [3286] Should we cover expect and/or Tk on our public Tcl courses? - (2011-05-11)
  [3572] Adding Expect on top of Tcl - what is it and where can I get a training course to learn about it? - (2012-01-08)
  [4678] Expect with Ruby - a training example to get you started - (2016-05-18)

Back to Which (virtual) host was visited? Tuning Apache log files, and Python analysis

Previous and next or Horse's mouth home

Forward to Fixing damaged MySQL tables - Error 1712 and Error 2013

Setting up and using a dict in Python - simple first example
Additional Python courses added to our schedule
Python - even named code blocks are objects
Fixing damaged MySQL tables - Error 1712 and Error 2013
Backup procedures - via backup server
Which (virtual) host was visited? Tuning Apache log files, and Python analysis
The unbalanced relationship between customer and provider
Finding sum, minimum, maximum and average in Python (and Ruby)
Selecting RECENT and POPULAR news and trends for your web site users