systemd backup and restore with practical examples

Introduction

Backups are only valuable when they run on time, finish safely, and can be restored quickly. Cron can schedule jobs, but production backup workflows also need dependency ordering, missed-run handling, centralized logging, failure notifications, and tight sandboxing. This is where systemd shines.

Important principle: systemd is not a backup application. It does not copy data or manage repositories. It orchestrates backup workflows by coordinating tools you already trust, such as rsync, Restic, BorgBackup, tar, database utilities, snapshot tools, and your own shell scripts.

The examples below show how to build reliable, auditable, and secure Linux backup automation with systemd services and timers.

1. systemd architecture for backup orchestration

At PID 1, systemd is the init system and service manager. Backups benefit from its unit model and state tracking.

• Units: typed definitions of things systemd manages. For backups we use:
• Service units: execute your scripts and backup tools
• Timer units: schedule service activations
• Target units: group related services (e.g., all backup jobs)
• Dependencies and ordering: Require=, Wants=, Before=, After= control correct sequencing
• State and restart logic: exit codes, Restart=, SuccessExitStatus= define health and retries
• Central logging: journald collects stdout/stderr for easy analysis

Diagram:

Application -> Files/DB
          ^        |
          |        v
   systemd service (backup.service)
          ^        |
          |        v
   systemd timer (backup.timer)
          |
    journald + notifications

2. Cron vs systemd timers

3. A production-ready backup service

This service runs a hardened rsync-based backup script as an unprivileged user. Adjust paths to your environment.

# /etc/systemd/system/backup.service
[Unit]
Description=Filesystem backup (rsync snapshots)
Documentation=man:systemd.service(5)
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
User=backup
Group=backup
WorkingDirectory=/var/lib/backup
EnvironmentFile=/etc/backup/backup.env
ExecStart=/usr/local/bin/backup-rsync.sh
# rsync returns 24 when files vanish during transfer; treat as success
SuccessExitStatus=0 24
TimeoutStartSec=2h
Restart=no

# Resource tuning
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=6

# Hardening
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/lib/backup /mnt/backup-target
CapabilityBoundingSet=
LockPersonality=true
RestrictRealtime=true
RestrictSUIDSGID=true
SystemCallArchitectures=native

[Install]
WantedBy=backup.target

Why harden backup jobs: backups routinely touch broad data sets and credentials. Sandboxing reduces blast radius if a script or tool misbehaves or is exploited.

4. A realistic rsync backup script

This example creates deduplicated snapshots with hard links, handles locking, enforces disk space checks, logs clearly, and performs basic retention.

# /usr/local/bin/backup-rsync.sh
#!/usr/bin/env bash
set -euo pipefail

: "${RSYNC_SRC:?RSYNC_SRC not set}"
: "${RSYNC_DST:?RSYNC_DST not set}"      # e.g. /mnt/backup-target/host1
RETENTION_DAYS="${RETENTION_DAYS:-30}"
MIN_FREE_GB="${MIN_FREE_GB:-10}"
SNAP_ROOT="$RSYNC_DST/snapshots"
CUR="$SNAP_ROOT/current"
TS="$(date +%F-%H%M%S)"
NEW="$SNAP_ROOT/$TS"

log() { echo "[backup] $(date -Is) $*"; }
notify_err() { logger -t backup "$*" || true; }

mkdir -p "$SNAP_ROOT"
mkdir -p /run/lock
exec 9>/run/lock/backup-rsync.lock
if ! flock -n 9; then log "Another backup is running"; exit 0; fi

# Disk space check
avail_gb=$(df -P "$RSYNC_DST" | awk 'NR==2 {print int($4/1024/1024)}')
if (( avail_gb < MIN_FREE_GB )); then
  notify_err "Insufficient free space: ${avail_gb}GB < ${MIN_FREE_GB}GB"
  exit 70
fi

log "Starting rsync snapshot to $NEW"
mkdir -p "$NEW.incomplete"
trap 'rm -rf "$NEW.incomplete"' EXIT

link_dest=( )
[[ -e "$CUR" ]] && link_dest=(--link-dest="$CUR")

set +e
/usr/bin/rsync -aHAX --numeric-ids \
  --delete-delay --partial --info=stats2,progress2 \
  --one-file-system "${link_dest[@]}" \
  "$RSYNC_SRC/" "$NEW.incomplete/"
rc=$?
set -e

if [[ $rc -ne 0 && $rc -ne 24 ]]; then
  notify_err "rsync exited with $rc"
  exit $rc
fi

mv "$NEW.incomplete" "$NEW"
ln -sfn "$NEW" "$CUR"

# Retention: delete snapshots older than N days
find "$SNAP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +"$RETENTION_DAYS" -not -name current -exec rm -rf {} +

log "Backup complete: $NEW"
exit 0

Example environment file:

# /etc/backup/backup.env
RSYNC_SRC=/srv/app-data
RSYNC_DST=/mnt/backup-target/host1
RETENTION_DAYS=30
MIN_FREE_GB=20

5. Scheduling with systemd timers

A daily run at 02:15, catch-up after downtime, and jitter to avoid thundering herds.

# /etc/systemd/system/backup.timer
[Unit]
Description=Schedule filesystem backup

[Timer]
OnCalendar=02:15
OnBootSec=15min
AccuracySec=1min
RandomizedDelaySec=20min
Persistent=true
Unit=backup.service

[Install]
WantedBy=timers.target

Other examples:

• Hourly: OnCalendar=hourly
• Every 6 hours: OnCalendar=--* 00,06,12,18:00:00
• From last activation: OnUnitActiveSec=24h

Enable and start: systemctl enable --now backup.timer

6. Monitoring and troubleshooting

• Status and last run: systemctl status backup.service
• View logs: journalctl -u backup.service -n 200 --since=yesterday
• List timers: systemctl list-timers --all
• Inspect failures: systemctl --failed; journalctl -xeu backup.service
• Show last exit code: systemctl show -p Result,ExecMainStatus backup.service

If a run was missed while the host was down, Persistent=true replays it on boot.

7. Coordinating application-aware backups

Some applications require quiescing or a logical dump. Example: stop PostgreSQL, back up data, restart, validate.

# /etc/systemd/system/backup-pg.service
[Unit]
Description=Application-aware backup: PostgreSQL + files
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
User=backup
Group=backup
EnvironmentFile=/etc/backup/pg.env
ExecStartPre=/bin/systemctl stop postgresql.service
ExecStart=/usr/local/bin/backup-rsync.sh
ExecStartPost=/bin/systemctl start postgresql.service
ExecStartPost=/usr/bin/pg_isready -q -t 30
TimeoutStartSec=2h
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/lib/backup /mnt/backup-target

[Install]
WantedBy=backup.target

Note: prefer logical dumps (pg_dump/pg_basebackup) or snapshot integration for zero downtime. This example demonstrates ordering with ExecStartPre/ExecStart/ExecStartPost.

8. Restic integration

Environment and credentials:

# /etc/restic/backup.env (0600)
RESTIC_REPOSITORY=s3:s3.example.com/restic/host1
RESTIC_PASSWORD_FILE=/etc/restic/pass
RESTIC_CACHE_DIR=/var/cache/restic
RESTIC_TAGS=host1,data

Service and timer:

# /etc/systemd/system/restic-backup.service
[Unit]
Description=Restic backup
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/backup.env
ExecStart=/usr/bin/restic backup /srv/app-data --tag ${RESTIC_TAGS} --one-file-system
ExecStartPost=/usr/bin/restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6
Nice=10
IOSchedulingClass=best-effort
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/var/cache/restic

[Install]
WantedBy=backup.target

# /etc/systemd/system/restic-backup.timer
[Unit]
Description=Schedule Restic backup

[Timer]
OnCalendar=03:00
Persistent=true
Unit=restic-backup.service

[Install]
WantedBy=timers.target

Verification and restore:

• Integrity: systemctl start restic-check.service with ExecStart=/usr/bin/restic check
• List snapshots: restic snapshots
• Restore latest: restic restore latest --target /restore

9. BorgBackup integration

# /etc/systemd/system/borg-backup.service
[Unit]
Description=BorgBackup job
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
Environment=BORG_REPO=/mnt/borg/host1 BORG_PASSCOMMAND="cat /etc/borg/pass"
ExecStart=/usr/bin/borg create --stats --compression zstd,6 ::host1-{now:%Y-%m-%d-%H%M} /srv/app-data
ExecStartPost=/usr/bin/borg prune --list --keep-daily 7 --keep-weekly 4 --keep-monthly 6
ExecStartPost=/usr/bin/borg compact
Nice=10
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=read-only
ReadWritePaths=/mnt/borg

[Install]
WantedBy=backup.target

Check and restore:

• borg list
• borg extract ::host1-2024-07-01-0215 path/in/archive -C /restore
• Mount for browsing: borg mount ::host1-latest /mnt/borgfs

10. Notifications and OnFailure handlers

Use OnFailure= to trigger alerts when a unit fails.

# /etc/systemd/system/backup.service (snippet)
OnFailure=notify-backup@%n.service

# /etc/systemd/system/[email protected]
[Unit]
Description=Notify on failure of %I

[Service]
Type=oneshot
Environment=WEBHOOK_URL=https://hooks.example/backup
ExecStart=/usr/bin/curl -fsS -X POST -H 'Content-Type: application/json' \
  -d '{"text":"Backup unit %I failed on %H"}' ${WEBHOOK_URL}

Swap curl for mailx, sendmail, Slack, or Discord webhooks as needed.

11. Security essentials

• Run as least-privilege users; avoid root when possible
• ProtectSystem=strict and allow only explicit ReadWritePaths
• ProtectHome=read-only or yes
• PrivateDevices=true, PrivateTmp=true, NoNewPrivileges=true
• Store secrets in root-owned 0600 files (EnvironmentFile=)
• Consider DynamicUser=yes with StateDirectory= for ephemeral accounts

12. Verify your backups

A backup without a tested restore should never be considered trustworthy.

• Periodic checks: restic check, borg check, rsync spot-verify
• Sample restore to a staging path and hash-compare critical files
• Boot a test VM and run full app validation against restored data

13. Restore automation

Coordinate restores with systemd to avoid partial rollbacks.

# /etc/systemd/system/[email protected]
[Unit]
Description=Restore application data for %i
After=network-online.target
Requires=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/backup.env
ExecStartPre=/bin/systemctl stop app@%i.service
ExecStart=/usr/bin/restic restore latest --target /restore/%i
ExecStartPost=/usr/bin/rsync -aHAX --delete /restore/%i/ /srv/%i/
ExecStartPost=/bin/systemctl start app@%i.service
ExecStartPost=/usr/bin/curl -fsS http://127.0.0.1:8080/healthz
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/restore /srv/%i

14. Disaster recovery example

• Provision a new server with OS, users, and network
• Mount or configure backup repository access (S3, NFS, disk)
• Install backup tools and copy credential files
• Restore configuration: system configs, unit files, env files
• Restore data: restic restore or borg extract to staging, then rsync to final paths
• Restore database: use pg_dump files or Borg/Restic snapshots of dumps
• Start services in order; validate health checks and logs
• Rotate DNS or load balancer back to the node
• Run repository checks and capture a DR report in tickets/wikis

15. Common mistakes

• Running all backups as root unnecessarily
• Forgetting Persistent=true, leading to missed jobs
• Cron and systemd both scheduling the same script
• No locking, causing overlapping runs
• Backing up live databases without quiescing or dumps
• No retention or verification policies
• Logs not centralized in journald or rotated poorly

16. Performance considerations

• Nice and IO scheduling to reduce impact: Nice=10, IOSchedulingClass=best-effort
• RandomizedDelaySec to spread load across fleets
• Limit parallelism to match IO bandwidth; schedule large jobs off-peak
• Use one-file-system flags to avoid crossing mounts unexpectedly
• Keep repositories on fast, reliable storage; monitor space proactively

17. Troubleshooting quick reference

• Timer never fires: systemctl list-timers; check calendar syntax; ensure timer enabled
• Service exits immediately: systemctl status ...; verify ExecStart path and permissions
• Permission denied: review sandboxing and ReadWritePaths
• Repository unavailable: test network and credentials; add Wants=network-online.target
• Mount failures: ensure mounts are Before= and RequiredBy= the service needing them
• Timeouts: raise TimeoutStartSec and check for slow disks or huge deltas

18. Best practices checklist

• Least privilege users and strict sandboxes
• Encrypted backups with protected keys
• Off-site or off-host storage
• Regular test restores and integrity checks
• Monitored timers and OnFailure notifications
• Documented runbooks and DR drills

19. End-to-end architecture

Application -> Data + DB
     |
     v
systemd services (backup, restic, borg)
     |
backup scripts -> tools (rsync/Restic/Borg)
     |
Encrypted repository (disk/S3/NFS)
     |
Off-site storage
     |
Restore server -> validate -> return to prod

Conclusion

systemd provides reliable orchestration for Linux backup workflows: scheduling with timers, safe execution with hardened services, dependency ordering, centralized logging, and actionable monitoring. Your backup software protects data; systemd coordinates when and how it runs, how failures surface, and how restores proceed. The most valuable backup is the one you have restored and validated successfully.