## Intro

Backup is not just copying files; it is the proven ability to restore the exact data, configuration, and state you need within a defined time. Different recovery goals (RPO/RTO) require different methods. In practice you will mix:

- File backups: user data, scripts, project directories.

- System backups: OS configuration, services, package lists, keys.

- Snapshots: fast point-in-time filesystem states (Btrfs/LVM/ZFS/Timeshift) used to create consistent backups, not as the only copy.

- Disk imaging: sector-level clones for bare-metal rebuilds.

- Application-aware backups: app data directories, secrets, manifests.

- Database backups: logical dumps or engine-native/consistent snapshots.

A backup that has never been restored is an unverified backup.

## A quick backup-selection framework

Answer these to choose an approach:

- What must you recover? Files only, full system, or bare metal.

- How much data loss is acceptable (RPO)? Minutes, hours, days.

- How fast must you be back (RTO)? Minutes, hours.

- Where will you store copies? Local, remote, off-site, immutable.

- How will you test restores and monitor failures?

Mapping common needs to tools:

- Small servers and laptops: rsync or Timeshift for system; Restic for encrypted off-site.

- Servers with changing datasets: Restic or Borg for deduplicated, versioned backups.

- Databases with strict RPO: engine-native dumps + binlog/WAL archiving; optionally snapshot-assisted.

- Bare metal recovery: Clonezilla/Rear, plus config and data backups for currency.

Follow the 3-2-1 rule: 3 copies, 2 media types, 1 off-site. Use immutable storage where feasible.

## Plan RPO and RTO

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Workload</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">RPO</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">RTO</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Primary</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Off-site</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Immutable</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Web app + DB</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">15m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">60m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Restic/Borg nightly + DB binlog/WAL</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">S3-compatible</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Object lock</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Engineering laptops</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">24h</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">24h</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Timeshift + Restic</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">NAS/cloud</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Optional</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Infrastructure hosts</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">4h</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">2h</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Borg + LVM/Btrfs snapshots</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Cloud</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Required</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Single database server</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">5m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">60m</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Logical dump + continuous logs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Remote object storage</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Object lock</td></tr></tbody>
</table>
</div>

## What to back up (and what to exclude)

Include:

- /home: user data, SSH configs, GPG, dotfiles.

- /etc: system config, network, fstab, DNS, sudoers.

- Keys and secrets: /etc/ssh, user ~/.ssh, app secrets (source of truth).

- systemd: /etc/systemd/system and overrides in /etc/systemd/system/*.d/.

- Cron: /var/spool/cron, /etc/cron.*, user crontabs.

- Docker/containers: compose files, .env, pinned images list, volumes.

- Kubernetes: manifests/Helm charts; etcd backup if self-managed.

- Databases: consistent logical dumps or engine-native snapshots.

- Config management: Ansible/Puppet/Chef repos.

- Package lists: dpkg --get-selections or rpm -qa.

- Logs (selectively): for compliance/forensics.

Usually exclude: /proc, /sys, /dev, /run, /tmp, caches, ephemeral containers, build artifacts. Prefer reinstalling packages instead of copying /usr binaries.

## Create backups: safe, repeatable commands

The examples below preserve permissions, ACLs, and xattrs where relevant. Run dry-runs first and send outputs to logs.

### Files and permissions with rsync

Key options: -a (archive), -A (ACLs), -X (xattrs), --numeric-ids, --delete (mirror deletions).

- Local mirror with ACLs/xattrs and a dry-run first:

# Preview
rsync -aAXn --delete /srv/data/ /backups/data/
# Execute once the preview looks safe
rsync -aAX --delete --info=stats2 --human-readable /srv/data/ /backups/data/ 
 WARNING: --delete removes files in the destination that are not in the source. Verify with --dry-run before executing.

- Remote push over SSH with stable IDs:

rsync -aAX --numeric-ids --delete -e "ssh -i /root/.ssh/backup" \
 /etc/ /var/www/ /home/ backup@backup-host:/tank/host1/ 

- Snapshot-like rotation with hard links:

 # Create newest snapshot using previous as link base
rsync -aAX --delete --link-dest=/backups/daily.1/ /srv/ /backups/daily.0/
# Rotate pointer cheaply (hard links)
cp -al /backups/daily.{0,1} 

### Preserve ACLs and extended attributes explicitly

 # Record ACLs and xattrs (in addition to rsync/tar that also carry them)
getfacl -R /etc > /backups/metadata/etc.acl
getfattr -R -d /etc > /backups/metadata/etc.xattr 

### Configuration snapshot of /etc and package list

 # Archive /etc with ACLs and xattrs
sudo tar --acls --xattrs -cpf /backups/system/etc-$(date +%F).tar -C / etc
# Debian/Ubuntu package list
dpkg --get-selections > /backups/system/dpkg-selections-$(date +%F).txt
# RHEL/Rocky/Alma list
rpm -qa --qf '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n' > /backups/system/rpm-list-$(date +%F).txt
# systemd units and timers
sudo systemctl list-unit-files --type=service --all > /backups/system/systemd-services.txt
sudo systemctl list-timers --all > /backups/system/systemd-timers.txt 

### Encrypted off-site backups with Restic

 export RESTIC_REPOSITORY="s3:https://s3.example.com/my-bucket/host1"
export RESTIC_PASSWORD="change-me"
export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...
restic init
# Create a snapshot
restic backup /etc /var/www /home --exclude-file=/root/restic-excludes.txt
# List and verify
restic snapshots
restic check
# Retention
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
# Restore to a safe temp directory (recommended)
restic restore latest --target /restore-test 
 WARNING: Restoring directly to / (root) can overwrite live files. Prefer restoring to /restore-test and then selectively copy back after validation.

### Deduplicated archives with Borg

export BORG_REPO=ssh://backup@backup-host:22/./repos/host1
export BORG_PASSPHRASE="change-me"
borg init --encryption=repokey $BORG_REPO
borg create --stats --progress ::{hostname}-{now:%F-%H%M} /etc /var/www /home
borg prune -v --list --keep-daily=7 --keep-weekly=4 --keep-monthly=6
borg check --verify-data
# Browse or extract
borg mount ::2024-07-01 /mnt/borg 

### Snapshots to capture consistent states

 Use snapshots to freeze fast-changing data and then back up the frozen view.

# LVM snapshot example (root VG/LV names vary)
sudo lvcreate -L 2G -s -n data_snap /dev/vg/data
# Mount the snapshot read-only, back it up, then remove it
sudo mount -o ro /dev/vg/data_snap /mnt/datasnap
rsync -aAX /mnt/datasnap/ /backups/data/
sudo umount /mnt/datasnap && sudo lvremove -f /dev/vg/data_snap 
 WARNING: Ensure the snapshot has sufficient size to hold changes during backup, or it may become invalid.

### Disk imaging (for bare metal)

- Prefer Clonezilla/Partclone for offline imaging that skips free space.

- Use dd only when you must capture exact layouts/bootloaders.

WARNING: dd can destroy data if source/destination are wrong. Double-check device names and never image a live, changing filesystem without a snapshot.

## Database backups (safe patterns)

Copying live database files is unsafe unless using engine-aware snapshots. Prefer logical dumps or engine-native tools and test the restore in isolation.

### MySQL/MariaDB

# Consistent dump without locking tables (InnoDB)
mysqldump --single-transaction --routines --triggers --events \
 -u backup -p mydb | gzip > /backups/mysql/mydb-$(date +%F).sql.gz

# Restore test into an isolated database name
zcat /backups/mysql/mydb-2024-07-01.sql.gz | mysql -u restore -p mydb_restore 
 Prerequisites: account with least-privileged dump rights; sufficient binlog retention if you plan point-in-time recovery.

### PostgreSQL

# Logical dump in custom format
sudo -u postgres pg_dump -Fc mydb > /backups/pg/mydb-$(date +%F).dump
# Restore test to a new database
dropdb --if-exists mydb_restore && createdb mydb_restore
sudo -u postgres pg_restore -d mydb_restore /backups/pg/mydb-2024-07-01.dump
# Physical base backup (ensure wal_level=replica)
sudo -u postgres pg_basebackup -D /backups/pg/base -Ft -z -P 

### MongoDB and Redis

 mongodump --db mydb --archive=/backups/mongo/mydb-$(date +%F).archive.gz --gzip

redis-cli SAVE
cp /var/lib/redis/dump.rdb /backups/redis/dump-$(date +%F).rdb 
 WARNING: For restores that overwrite existing databases, schedule maintenance, stop services, and take a safety snapshot or copy first.

## Containers and volumes (Docker)

# Pin image versions for rebuilds
docker images --digests > /backups/docker/images-$(date +%F).txt
# Backup a named volume to a tar on the host
docker run --rm -v myvol:/data -v /backups:/backups alpine \
 sh -c "tar czf /backups/myvol-$(date +%F).tar.gz -C /data ."
# Restore into a fresh volume (non-destructive to existing)
docker volume create myvol
docker run --rm -v myvol:/data -v /backups:/backups alpine \
 sh -c "tar xzf /backups/myvol-2024-07-01.tar.gz -C /data" 
 Store secrets in a dedicated system (Docker secrets, Vault) and back up their source of truth.

## Automate and monitor

Use systemd timers or cron. Report success/failure to monitoring.

# Nightly Restic with a health check ping
0 2 * * * RESTIC_REPOSITORY=... RESTIC_PASSWORD=... \
 /usr/local/bin/restic backup /etc /var/www /home \
 && curl -fsS -m 10 https://hc-ping.com/UUID || echo "backup failed" >&2 
 Rotate and prune (restic forget --prune, borg prune). Watch exit codes and logs.

## Test restores and verify integrity

Clearly separate backup creation from restore testing. Never test by overwriting production paths.

Restore-verification checklist:

- Select a recent snapshot and a random older snapshot.

- Restore to a temporary directory or isolated instance.

- Validate file counts, permissions, ACLs, xattrs (diff -r, getfacl/getfattr).

- For configs, run syntax checks (e.g., nginx -t, systemd-analyze verify).

- For databases, run a test restore and a basic query set.

- Run restic check or borg check and review logs.

- Document duration (RTO) and the recovery point (RPO achieved).

Example file restore test:

restic restore latest --target /restore-test
sudo rsync -aAXn /restore-test/etc/ /etc/ # dry-run compare 
 Example config promote (after validation):

# Prereqs: maintenance window, service stopped, local safety copy
sudo systemctl stop nginx
sudo cp -a /etc/nginx /etc/nginx.bak-$(date +%s)
sudo rsync -aAX /restore-test/etc/nginx/ /etc/nginx/
sudo nginx -t && sudo systemctl start nginx 

## Failure-triage runbook (backup or restore)

- Scope the issue: which host, which dataset, since when, one-off vs recurring.

- Check credentials and connectivity: DNS, network, firewall, SSH/HTTPS keys.

- Inspect repository health: restic check / borg check; available space and object-lock status.

- Validate source filesystem health: dmesg, SMART, fsck needs, inode exhaustion.

- Review recent changes: package upgrades, kernel, permissions, SELinux/AppArmor.

- Retry safely with increased verbosity; capture logs.

- Escalate with artifacts: command, versions, logs, snapshot IDs, timing, size.

## Common pitfalls

- Relying only on /home; ignoring /etc, databases, and secrets.

- Never testing restores or mixing testing with production paths.

- Backups on the same disk/NAS without immutability or off-site copies.

- Losing ACLs/xattrs/ownership; applications then fail at runtime.

- Using dd for versioned backups; it is slow to manage and not incremental.

- No retention/rotation; either buckets fill or needed history is gone.

- No monitoring; failures stay silent for months.

## Conclusion

 Linux backup and restore is about recovery, not copying. Start with RPO/RTO, back up the right data (including configuration and databases), encrypt and ship off-site copies, and automate with monitoring. Use snapshots to take consistent points, but always verify by restoring into safe locations and validating permissions and functionality. The only trustworthy backup is one you have restored and checked.