systemd Basic Commands with Practical Examples: A Hands-On Guide for Operators

Intro

Systemd is the default init system and service manager on most modern Linux distributions, including Ubuntu, Debian, Fedora, CentOS/RHEL, and Arch. As an operator, developer, or DevOps consultant, you will frequently need to start, stop, enable, inspect, and troubleshoot services. This guide covers the essential systemd commands with practical examples, focusing on real-world scenarios. You will learn how to check the systemd version, manage services with systemctl, inspect logs with journalctl, edit unit files safely, and recover from common failures. Each section includes commands with expected output and verification steps.

We emphasize operational safety: observe before changing, limit the blast radius, use placeholders instead of real secrets, verify results, and document recovery paths. By the end, you will have a solid command-line toolkit for daily systemd operations.

Version and Environment Inventory

Before running any systemd commands, confirm the environment. Check the systemd version, the operating system release, and the kernel version. This ensures you use the correct syntax and features.

Check systemd version:

systemctl --version

Expected output (example):

systemd 252 (252.5-1ubuntu3)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

Check OS release:

cat /etc/os-release

Expected output snippet:

NAME="Ubuntu"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 22.04.3 LTS"

Check kernel version:

uname -r

Expected output: 5.15.0-91-generic

Read-only observation: Use systemctl status (without sudo) to view the state of a specific service, for example, ssh:

systemctl status ssh

Expected output (trimmed):

● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-01-01 12:00:00 UTC; 2h 30min ago
       Docs: man:sshd(8)
             man:sshd_config(5)
   Main PID: 1234 (sshd)
      Tasks: 1 (limit: 4915)
     Memory: 1.2M
        CPU: 123ms
     CGroup: /system.slice/ssh.service
             └─1234 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

If the service is not running, the Active line will show inactive (dead) or failed. Note the exact name of the service unit, as it is case-sensitive and may differ across distributions.

Prerequisites: Most management commands (start, stop, restart, enable, disable) require root privileges. Use sudo or run as root. Always verify your user has the necessary permissions via sudo -l.

Safe Configuration Path

Editing systemd unit files directly can be risky. Use the correct tools and methods to avoid breaking services.

Viewing Unit Files

List all unit files:

systemctl list-unit-files --type=service

Expected output (first few lines):

UNIT FILE                                  STATE           VENDOR PRESET
accounts-daemon.service                    enabled         enabled
acpid.service                              disabled        enabled
alsa-restore.service                       static          enabled
...

Display the contents of a unit file:

systemctl cat ssh.service

Expected output:

# /lib/systemd/system/ssh.service
[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=sshd.service

Creating Override Files

Never edit files in /lib/systemd/system/ or /usr/lib/systemd/system/ directly, as package updates may overwrite them. Instead, create drop-in override files.

Create an override directory and edit the override:

sudo systemctl edit ssh.service

This opens a text editor (usually nano or vim) for a new file located at /etc/systemd/system/ssh.service.d/override.conf. Add directives under the appropriate section, for example, to change the restart policy:

[Service]
Restart=always
RestartSec=5s

Save and exit. Then reload the systemd manager configuration and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ssh.service

Verify the override took effect:

systemctl cat ssh.service

You should see additional lines showing the drop-in file.

Check for syntax errors before applying:

sudo systemd-analyze verify ssh.service

If there are errors, it will report them; otherwise, no output indicates success.

Blast radius: Only the affected service is modified. Use systemctl status ssh.service to confirm it is running after the change. If the service fails to start, revert the override by deleting the file and running sudo systemctl daemon-reload and sudo systemctl restart ssh.service.

Recovery path: Always keep a copy of the original unit file or know how to regenerate it. For overrides, simply removing the drop-in file restores the default behavior.

Verification and Diagnostics

Verification ensures services are functioning correctly. Use a combination of status, logs, and dependency checks.

Service Status and Health

Check if a service is active (running) and enabled at boot:

systemctl is-active ssh.service
systemctl is-enabled ssh.service

Expected output: active and enabled, respectively. If not, the commands return inactive or failed, and disabled or static, with a non-zero exit code.

View detailed status including recent log lines:

systemctl status ssh.service --no-pager -l

This shows the last few log entries from the service, which can be invaluable for diagnosing issues.

Analyzing Boot and Dependency Issues

Show the boot process timeline to identify slow services:

systemd-analyze blame

Expected output (example):

5.123s NetworkManager-wait-online.service
2.456s apt-daily.service
1.789s docker.service
...

Show the critical chain for a specific target:

systemd-analyze critical-chain multi-user.target

This helps understand which services block the boot process.

List dependencies of a service:

systemctl list-dependencies ssh.service

Expected output:

ssh.service
● ├─system.slice
● ├─sshd-keygen.service
● └─network.target
●   └─network-pre.target

This shows what ssh.service requires and what depends on it.

Journal Inspection

The journal is the central log for systemd. Use journalctl to query logs.

View logs for a specific service since the last boot:

journalctl -u ssh.service -b

Follow logs in real time:

journalctl -u ssh.service -f

Filter by priority (e.g., errors and worse):

journalctl -u ssh.service -p err

Limit output to a time range:

journalctl -u ssh.service --since "2024-01-01 10:00:00" --until "2024-01-01 12:00:00"

Check disk usage of the journal:

journalctl --disk-usage

Expected output: Archived and active journals take up 152.0M in the file system.

If the journal is too large, configure limits in /etc/systemd/journald.conf (e.g., SystemMaxUse=500M) and restart systemd-journald.

Failure Modes and Recovery

Services fail for various reasons: configuration errors, missing dependencies, resource limits, or crashes. Knowing how to identify and recover is crucial.

Common Failure States

• inactive (dead): Service is not running and not scheduled to run.
• failed: Service attempted to start but crashed or exited with an error.
• activating: Service is in the process of starting.
• deactivating: Service is in the process of stopping.
• auto-restart: Service will be restarted automatically by systemd.

Check for failed units system-wide:

systemctl --failed

Expected output (if any):

  UNIT                         LOAD   ACTIVE SUB    DESCRIPTION
● myapp.service                loaded failed failed My Custom Application

If there are failures, investigate with systemctl status myapp.service and journalctl -u myapp.service -b -e (the -e jumps to the end of the log).

Debugging a Failed Service

Suppose a service myapp.service fails. Perform the following:

1. Check status:

   systemctl status myapp.service

Look for the Active line and any error messages.

1. View recent logs:

   journalctl -u myapp.service -n 50 --no-pager

This shows the last 50 log lines.

1. Examine the unit file:

   systemctl cat myapp.service

Check the ExecStart path and command syntax.

1. Run the ExecStart command manually (if safe) to see errors directly.

1. Check file permissions and paths. Ensure the executable and configuration files are accessible to the user specified in the unit (User= directive).

1. Use systemd-analyze verify to check the unit file syntax:

   sudo systemd-analyze verify myapp.service

Recovery Actions

Restart a failed service:

sudo systemctl restart myapp.service

If the service has a dependency issue, restart dependencies as well:

sudo systemctl restart myapp.service --no-block

Reset failed state:

sudo systemctl reset-failed myapp.service

Disable a service temporarily:

sudo systemctl stop myapp.service
sudo systemctl disable myapp.service

Mask a service to prevent it from starting:

sudo systemctl mask myapp.service

To unmask:

sudo systemctl unmask myapp.service

Rollback an override:

If a recent override caused the failure, delete the drop-in file:

sudo rm /etc/systemd/system/myapp.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart myapp.service

Recovering from Boot Failures

If the system fails to boot due to a bad service, use recovery mode or a live CD. Add systemd.unit=rescue.target or systemd.unit=emergency.target to the kernel command line to boot into a minimal environment. Then disable or fix the offending service.

Operations Checklist

Use this checklist to safely manage systemd services in production.

Before Any Change

• [ ] Confirm the exact service name: systemctl list-units --type=service | grep <pattern>
• [ ] Record current state: systemctl status <service> --no-pager > /tmp/service_status_before.txt
• [ ] Check dependencies: systemctl list-dependencies <service>
• [ ] Ensure you have a rollback plan (e.g., backup of unit file overrides or documented original state).
• [ ] Notify affected users or team if the service is critical.

During Change

• [ ] Use sudo systemctl <action> <service> rather than editing files directly when possible.
• [ ] For unit file edits, use sudo systemctl edit <service> to create overrides.
• [ ] After editing, run sudo systemd-analyze verify <service> to catch syntax errors.
• [ ] Reload daemon if unit files changed: sudo systemctl daemon-reload.
• [ ] Apply the change: sudo systemctl restart <service> or sudo systemctl reload <service> if supported.

After Change

• [ ] Verify service is active and running: systemctl is-active <service> should return active.
• [ ] Check for errors in logs: journalctl -u <service> -n 20 --no-pager.
• [ ] Confirm the service is enabled if it should start at boot: systemctl is-enabled <service>.
• [ ] Monitor for a few minutes to ensure no unexpected restarts: systemctl status <service> shows the uptime.
• [ ] Record the change and any new observations for future reference.

Example: Restarting Nginx

# Before
systemctl status nginx --no-pager | tee /tmp/nginx_status_before.txt

# During
sudo systemctl restart nginx

# After
systemctl is-active nginx   # expect: active
journalctl -u nginx -n 20 --no-pager

Conclusion

Mastering systemd basic commands is essential for anyone managing Linux systems. This guide covered inventory, safe configuration, verification, failure recovery, and an operations checklist. By following the principles of observation, minimal intervention, and verification, you can operate systemd-managed services with confidence.

As a next step, practice these commands on a test system. Create a simple service, make changes using overrides, induce failures, and recover. The more familiar you are with systemctl, journalctl, and systemd-analyze, the faster you can diagnose and fix issues in production. Remember to always have a rollback plan and document your actions.