## Intro

Upgrading systemd is routine on modern Linux, but it touches PID 1, service supervision, logging, and boot ordering. A careless change can strand a host in emergency mode or break critical services. This guide shows a practical, low-drama path to upgrade or migrate to newer systemd versions, validate behavior, and recover fast if something misbehaves. You will get a clear workflow, a safe local pilot, hands-on commands, and concrete examples for services and timers.

## Workflow Overview

Follow these stages to make upgrades predictable:

- Assess : inventory services, timers, drop-ins, and custom scripts. Identify anything nonstandard.

- Prepare safety : capture OS and package info, enable persistent journals, prepare snapshots and rollback commands.

- Pilot : run one small, measurable upgrade task on a noncritical host or VM.

- Stage and upgrade : update systemd packages and related libraries, then plan a controlled reboot window.

- Validate : confirm boot targets, services, timers, and critical-chain timing. Inspect logs for regressions.

- Migrate units : convert legacy scripts or refine unit types and sandboxing using examples below.

- Observe : monitor for 24–72 hours. Roll back quickly if you detect regressions.

## Preparation Checklist

Run commands on a test host first, then on production during a window with console access.

### Baseline and inventory

- Record current versions:

systemctl --version
cat /etc/os-release 

- List enabled units and overrides:

 systemctl list-unit-files --type=service --state=enabled
systemd-delta 

- Inventory timers and cron jobs:

 systemctl list-timers --all
crontab -l
sudo ls -1 /etc/cron.* 

### Journals and logging

- Ensure persistent journaling so you can read logs after reboot:

 sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal 

### Console and access

- Ensure you have out-of-band access (serial/IPMI/tty) in case SSH does not come up.

- Keep a root shell open on the console or a tmux session; avoid disconnecting mid-upgrade.

### Snapshots and downgrade path

- Filesystem snapshot recommendation:

- LVM: sudo lvcreate -s -n root-pre-sd -L 10G /dev/vg/root

- Btrfs: sudo btrfs subvolume snapshot -r / @ /@snapshots/@pre-systemd

- Package rollback prep:

- Debian/Ubuntu: sudo apt-cache policy systemd ; note the candidate and currently installed version. Optionally apt download systemd systemd-sysv libsystemd0

- RHEL/Fedora: sudo dnf history list systemd ; note the last known-good NVR

### Change window

- Schedule a reboot window even if not strictly required. Reboots surface ordering issues you want to validate.

## Local Pilot Plan

 Start with one host or VM and one measurable goal. Example pilot: upgrade systemd, convert one cron to a timer, and verify boot and service health.

### Pilot scope

- Host : a noncritical VM that mirrors production units.

- Change 1 : upgrade systemd to the target version.

- Change 2 : convert a small daily cron job to a systemd timer.

- Success criteria :

- Host reboots cleanly to default target.

- 0 failed units ( systemctl --failed ).

- Timer triggers and completes once with exit code 0.

- No new errors at priority err or higher in journalctl -b -p err..alert .

### Pilot commands

- Validate baseline:

systemctl --version
systemctl list-units --failed
journalctl -b -p warning..alert --no-pager | tail -n +1 

- After upgrade and reboot, rerun the same checks and compare.

## Upgrade Execution

### Debian/Ubuntu (apt)

- Update metadata and review versions:

 sudo apt-get update
apt-cache policy systemd 

- Upgrade only systemd first to limit blast radius:

 sudo apt-get install --yes systemd libsystemd0 

- Plan a controlled reboot window:

 sudo systemctl reboot 

### RHEL/CentOS/Alma/Rocky/Fedora (dnf)

- Refresh metadata and review versions:

 sudo dnf --refresh info systemd 

- Upgrade systemd and libraries:

 sudo dnf upgrade -y systemd systemd-libs 

- Reboot during the change window:

 sudo systemctl reboot 

### Post-upgrade hygiene

- Reload unit files to pick up any packaging changes:

 sudo systemctl daemon-reload 

- Clear any stale unit state if needed:

 sudo systemctl reset-failed 

## Validation

### Boot and targets

- Confirm default target and reachability:

 systemctl get-default
systemctl is-system-running 

- Inspect the critical chain for slow or blocked services:

 systemd-analyze critical-chain 

### Services and sockets

- Look for failures and degraded state:

 systemctl --failed 

- Validate key services (example: nginx):

 systemctl status nginx
sudo journalctl -u nginx -b --no-pager | tail -50 

### Journals and errors

- Scan high-priority messages since boot:

 journalctl -b -p err..alert --no-pager 

### Timers and tasks

- List and verify timers:

 systemctl list-timers 

- Trigger a timer early for testing:

 sudo systemctl start mybackup.timer
sudo systemctl status mybackup.service 

### Unit verification

- Statically verify unit files for common issues:

 sudo systemd-analyze verify /etc/systemd/system/*.service 

### Network and remote access

- Confirm network online target and multi-user reachability:

 systemctl status NetworkManager.service | head -20
systemctl status systemd-networkd.service | head -20 
 If anything looks off, capture logs and proceed to rollback or remediation before expanding rollout.

## Migration Examples

### Example A: Convert a SysV-style app to a native service

Suppose you had /etc/init.d/myapp managing a simple binary. Replace it with a native unit at /etc/systemd/system/myapp.service :

[Unit]
Description=MyApp REST API
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
EnvironmentFile=-/etc/myapp/env
ExecStart=/opt/myapp/bin/myapp --port=8080
Restart=on-failure
RestartSec=2s
# Resource and security hardening
LimitNOFILE=65536
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
AmbientCapabilities=

[Install]
WantedBy=multi-user.target 
 Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now myapp.service 
 Validate:

systemctl status myapp
journalctl -u myapp -b --no-pager | tail -100 

### Example B: Add a drop-in to tune an existing service (nginx)

 Create a drop-in to raise file descriptor limit without editing vendor unit:

sudo systemctl edit nginx 
 This creates /etc/systemd/system/nginx.service.d/override.conf with:

[Service]
LimitNOFILE=131072 
 Apply and validate:

sudo systemctl daemon-reload
sudo systemctl restart nginx
systemctl show nginx -p LimitNOFILE 

### Example C: Convert a cron job to a timer

 Original cron (daily at 02:15): backs up /srv to /backups .

Create service /etc/systemd/system/backup.service :

[Unit]
Description=Nightly backup

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup.sh /srv /backups
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=6 
 Create timer /etc/systemd/system/backup.timer :

[Unit]
Description=Run nightly backup at 02:15

[Timer]
OnCalendar=*-*-* 02:15:00
RandomizedDelaySec=5m
Persistent=true

[Install]
WantedBy=timers.target 
 Enable and test:

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers | grep backup
sudo systemctl start backup.service
systemctl status backup.service 

### Example D: Verify and tighten units after upgrade

- Check for deltas and deprecated directives:

 systemd-delta
sudo systemd-analyze verify /etc/systemd/system/*.service 

- Consider Type=notify for services that signal readiness, or add ExecStartPre checks to fail fast.

## Rollback and Recovery

### Fast rollback decisions

- If the host boots but key services fail: downgrade packages and revert unit changes.

- If the host cannot reach multi-user: boot into rescue, revert from snapshot or downgrade systemd.

### Boot into rescue or emergency

 At bootloader, edit kernel command line to include one of:

- systemd.unit=rescue.target

- systemd.unit=emergency.target

Once in rescue shell, inspect logs:

journalctl -xb --no-pager | less 

### Package downgrade

 Debian/Ubuntu:

apt-cache madison systemd
sudo apt-get install systemd=VERSION libsystemd0=VERSION
sudo apt-mark hold systemd libsystemd0 # optional 
 RHEL/Fedora:

sudo dnf history systemd
sudo dnf downgrade -y systemd systemd-libs 

### Snapshot restore

 LVM:

sudo lvconvert --merge /dev/vg/root-pre-sd
sudo reboot 
 Btrfs:

- Mount snapshot and restore or use a subvolume rollback strategy appropriate for your layout.

### Unit change rollback

- Remove or rename drop-ins under /etc/systemd/system/*.d

- Restore prior unit files from backup, then:

sudo systemctl daemon-reload
sudo systemctl restart affected.service 

### Post-rollback

 Document the root cause and adjust the validation checklist before the next attempt.

## Conclusion

Treat systemd upgrades and migrations as an incremental, testable change, not a leap of faith. Inventory first, prepare a rollback, run a narrow pilot, then upgrade, validate, and monitor before broad rollout. Use native units and timers with drop-ins to bring custom services under reliable supervision. Keep your rescue and downgrade paths documented and tested. Next steps: run the pilot on a noncritical host this week, convert one cron job to a timer, and capture a short runbook for your production change window.