This week I finally got around to writing a new backup script for my webserver. I have it automatically pushing backups to a device at home, but in the past I’d only been doing a nightly rsync with --delete and periodic offline backups. The problem with this was that should something happen to my server and cause a data loss, but not be noticed before the next backup ran, the current backup would be modified reflect the now-compromised data, potentially causing massive data loss. Clearly this was a bad thing, and something had to be done.
A new backup scheme was devised and now that the new scripts are tweaked I wanted to present them here. rsync is still being used, but thanks to its glorious --link-dest option which makes hard links as it can, files already stored on disk (say, from a previous version of the backup) are reused, saving space. This is how Apple's Time Machine works, just without the nice GUI. The result is that I have a series of directories starting with backup.0 up through potentially backup.30 on the target, each containing a different backup. The suffixed number represents how many versions old the backup is. These versions are generally created once per day, but on days where the backup does not complete successfully the version is not incremented.
To start, there is a script called dailybackup.sh which runs once per day on banstyle.nuxx.net. This script pushes a backup to a Mac at home as follows:
- If needed, remotely execute rotatebackup.sh on the backup server. This will move backup.0 to backup.1, backup.1 to backup.2, keeping no more than 30 backups. The need to rotate backups is determined by the presence of backup.0/backup_complete. If there is no backup_complete file we know that the previous backup was not successful and that we should reuse backup.0.
- Create a new backup.0 and populate it with a backup_started flag file.
- Run the backup job via rsync.
- If the job completes successfully (exits with something other than 0 or 24), continue. Exit code 24 indicates that some files disappeared during backup, and as mail files (amongst others) tend to move and be deleted by users during the backup job, this is not a critical error for us.
- Remove backup_started and create the backup_complete flag.
Copies of the aforementioned scripts can be found here, if you’d like to look at / use them: dailybackup.sh · rotatebackups.sh
These scripts assume the presence of backup.0, a full copy of your backup, which you’ll have to create yourself before use. There’s also likely some necessary changes for your environment, most likely in some of the variables set at the top of the scripts, such as the number of days for which to keep backups and logs, the target hostname, SSH port, username, etc.
