A backup strategy that works: 3-2-1 for your server

A backup strategy is judged on the day you need it. This guide gives you a plan based on the 3-2-1 rule, shows how to back up files and databases on a Linux server with encrypted off-site copies, and how to prove that a restore works. A Windows section follows.

The 3-2-1 rule

• 3 copies of your data: the live data and two backups.
• 2 different kinds of storage, so one failure cannot destroy everything.
• 1 copy off-site, away from the server and its platform.

Many teams extend it to 3-2-1-1-0: one copy that cannot be changed or deleted (immutable or offline), and 0 errors in your restore tests.

Before you start

Write down two numbers for each system:

• How much data can you lose? If the server died now, how many hours of orders, posts or uploads would be acceptable to lose? This sets how often you back up.
• How quickly must it be back? This decides whether a restore from backup is fast enough, or whether you need a standby.

Step 1: Decide what to back up

Do not rely on copying a running database's files: they may be inconsistent. Dump them first.

Step 2: Dump the databases

Write dumps to a local folder that the file backup then picks up:

sudo mkdir -p /var/backups/db
sudo mysqldump --all-databases --single-transaction --routines --triggers | gzip | sudo tee /var/backups/db/all.sql.gz > /dev/null

sudo mkdir -p /var/backups/db
sudo mariadb-dump --all-databases --single-transaction --routines --triggers | gzip | sudo tee /var/backups/db/all.sql.gz > /dev/null

sudo mkdir -p /var/backups/db
sudo -u postgres pg_dump -Fc mydb | sudo tee /var/backups/db/mydb.dump > /dev/null

--single-transaction gives a consistent copy of InnoDB tables without locking your site. Replace mydb with your PostgreSQL database name.

Step 3: Back up off-site with restic

restic makes encrypted, deduplicated backups to many targets: another server over SFTP, an S3-compatible bucket and more. Install it from your distribution:

sudo apt install restic

Store the repository password in a root-only file, then create the repository. The example uses another server over SFTP, which needs SSH key access for root to the backup account there (see SSH keys); replace the address and path:

sudo install -m 600 /dev/null /root/.restic-password
sudo nano /root/.restic-password
sudo restic -r sftp:backup@198.51.100.20:/srv/restic/web1 --password-file /root/.restic-password init

Run a first backup:

sudo restic -r sftp:backup@198.51.100.20:/srv/restic/web1 --password-file /root/.restic-password backup /etc /var/www /var/backups/db

Verify: list the snapshots in the repository:

sudo restic -r sftp:backup@198.51.100.20:/srv/restic/web1 --password-file /root/.restic-password snapshots

Step 4: Schedule it

Put the dump and the backup in a script, /usr/local/bin/backup.sh, make it executable (sudo chmod 700 /usr/local/bin/backup.sh), and run it with a systemd timer. The service:

[Unit]
Description=Server backup

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

Save it as /etc/systemd/system/backup.service. The timer, /etc/systemd/system/backup.timer:

[Unit]
Description=Run the server backup every night

[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true

[Install]
WantedBy=timers.target

Enable it:

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

Verify: systemctl list-timers backup.timer shows the next run, and sudo journalctl -u backup.service shows the output of the last one.

Step 5: Keep generations and check the repository

Tell restic which snapshots to keep and remove the rest:

sudo restic -r sftp:backup@198.51.100.20:/srv/restic/web1 --password-file /root/.restic-password forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
sudo restic -r sftp:backup@198.51.100.20:/srv/restic/web1 --password-file /root/.restic-password check

Add both lines to the end of your backup script, so retention runs automatically.

Step 6: Test a restore

A restore test is the only proof. At least once a month, restore to a temporary folder or a test server:

sudo restic -r sftp:backup@198.51.100.20:/srv/restic/web1 --password-file /root/.restic-password restore latest --target /tmp/restore-test

Compare a few files, import a database dump into a test database and open the restored site. Note how long it took: that is your real recovery time.

Use the plan's backups and snapshots too

• VPS, VDS and web hosting plans include free weekly and monthly backups. Ask support if you need a restore from them.
• Windows VPS plans include free snapshots; take one before updates or risky changes. Snapshots complement off-site backups.
• On hosting plans, download your own copy from the control panel's backup tool regularly.

Windows Server

Install Windows Server Backup and back up the system and data to a second disk or network share:

Install-WindowsFeature Windows-Server-Backup
wbadmin start backup -backupTarget:E: -include:C: -allCritical -quiet

Copy SQL Server backups and important folders off the server as well, and test a restore regularly.

Troubleshooting

restic: "repository does not exist". The path or user in the -r value is wrong, or init was not run. Check SSH access with ssh [email protected].

Backups fill the disk. Dumps stay on the server after each run. Keep only the latest locally and rely on restic for history; see disk full.

The timer never runs. Check systemctl status backup.timer and that you ran systemctl daemon-reload after creating the files.

Next steps

• Recover after an incident: what to do if your server is hacked.
• Start over cleanly: reinstall the operating system.