## Intro Systemd is the default init system and service manager on most modern Linux distributions, including Ubuntu, Debian, Fedora, CentOS/RHEL, and Arch. As an operator, developer, or DevOps consultant, you will frequently need to start, stop, enable, inspect, and troubleshoot services. This guide covers the essential systemd commands with practical examples, focusing on real-world scenarios. You will learn how to check the systemd version, manage services with systemctl, inspect logs with journalctl, edit unit files safely, and recover from common failures. Each section includes commands with expected output and verification steps. We emphasize operational safety: observe before changing, limit the blast radius, use placeholders instead of real secrets, verify results, and document recovery paths. By the end, you will have a solid command-line toolkit for daily systemd operations. ## Version and Environment Inventory Before running any systemd commands, confirm the environment. Check the systemd version, the operating system release, and the kernel version. This ensures you use the correct syntax and features. Check systemd version: systemctl --version Expected output (example): systemd 252 (252.5-1ubuntu3) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE -TPM2 -BZIP2 -LZ4 -XZ -ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified Check OS release: cat /etc/os-release Expected output snippet: NAME="Ubuntu" VERSION="22.04.3 LTS (Jammy Jellyfish)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 22.04.3 LTS" Check kernel version: uname -r Expected output: 5.15.0-91-generic Read-only observation: Use systemctl status (without sudo) to view the state of a specific service, for example, ssh : systemctl status ssh Expected output (trimmed): ● ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-01-01 12:00:00 UTC; 2h 30min ago Docs: man:sshd(8) man:sshd_config(5) Main PID: 1234 (sshd) Tasks: 1 (limit: 4915) Memory: 1.2M CPU: 123ms CGroup: /system.slice/ssh.service └─1234 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups" If the service is not running, the Active line will show inactive (dead) or failed . Note the exact name of the service unit, as it is case-sensitive and may differ across distributions. Prerequisites: Most management commands ( start , stop , restart , enable , disable ) require root privileges. Use sudo or run as root. Always verify your user has the necessary permissions via sudo -l . ## Safe Configuration Path Editing systemd unit files directly can be risky. Use the correct tools and methods to avoid breaking services. ### Viewing Unit Files List all unit files: systemctl list-unit-files --type=service Expected output (first few lines): UNIT FILE STATE VENDOR PRESET accounts-daemon.service enabled enabled acpid.service disabled enabled alsa-restore.service static enabled ... Display the contents of a unit file: systemctl cat ssh.service Expected output: # /lib/systemd/system/ssh.service [Unit] Description=OpenBSD Secure Shell server Documentation=man:sshd(8) man:sshd_config(5) After=network.target auditd.service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] EnvironmentFile=-/etc/default/ssh ExecStartPre=/usr/sbin/sshd -t ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/usr/sbin/sshd -t ExecReload=/bin/kill -HUP $MAINPID KillMode=process Restart=on-failure RestartPreventExitStatus=255 Type=notify RuntimeDirectory=sshd RuntimeDirectoryMode=0755 [Install] WantedBy=multi-user.target Alias=sshd.service ### Creating Override Files Never edit files in /lib/systemd/system/ or /usr/lib/systemd/system/ directly, as package updates may overwrite them. Instead, create drop-in override files. Create an override directory and edit the override: sudo systemctl edit ssh.service This opens a text editor (usually nano or vim) for a new file located at /etc/systemd/system/ssh.service.d/override.conf . Add directives under the appropriate section, for example, to change the restart policy: [Service] Restart=always RestartSec=5s Save and exit. Then reload the systemd manager configuration and restart the service: sudo systemctl daemon-reload sudo systemctl restart ssh.service Verify the override took effect: systemctl cat ssh.service You should see additional lines showing the drop-in file. Check for syntax errors before applying: sudo systemd-analyze verify ssh.service If there are errors, it will report them; otherwise, no output indicates success. Blast radius: Only the affected service is modified. Use systemctl status ssh.service to confirm it is running after the change. If the service fails to start, revert the override by deleting the file and running sudo systemctl daemon-reload and sudo systemctl restart ssh.service . Recovery path: Always keep a copy of the original unit file or know how to regenerate it. For overrides, simply removing the drop-in file restores the default behavior. ## Verification and Diagnostics Verification ensures services are functioning correctly. Use a combination of status, logs, and dependency checks. ### Service Status and Health Check if a service is active (running) and enabled at boot: systemctl is-active ssh.service systemctl is-enabled ssh.service Expected output: active and enabled , respectively. If not, the commands return inactive or failed , and disabled or static , with a non-zero exit code. View detailed status including recent log lines: systemctl status ssh.service --no-pager -l This shows the last few log entries from the service, which can be invaluable for diagnosing issues. ### Analyzing Boot and Dependency Issues Show the boot process timeline to identify slow services: systemd-analyze blame Expected output (example): 5.123s NetworkManager-wait-online.service 2.456s apt-daily.service 1.789s docker.service ... Show the critical chain for a specific target: systemd-analyze critical-chain multi-user.target This helps understand which services block the boot process. List dependencies of a service: systemctl list-dependencies ssh.service Expected output: ssh.service ● ├─system.slice ● ├─sshd-keygen.service ● └─network.target ● └─network-pre.target This shows what ssh.service requires and what depends on it. ### Journal Inspection The journal is the central log for systemd. Use journalctl to query logs. View logs for a specific service since the last boot: journalctl -u ssh.service -b Follow logs in real time: journalctl -u ssh.service -f Filter by priority (e.g., errors and worse): journalctl -u ssh.service -p err Limit output to a time range: journalctl -u ssh.service --since "2024-01-01 10:00:00" --until "2024-01-01 12:00:00" Check disk usage of the journal: journalctl --disk-usage Expected output: Archived and active journals take up 152.0M in the file system. If the journal is too large, configure limits in /etc/systemd/journald.conf (e.g., SystemMaxUse=500M ) and restart systemd-journald . ## Failure Modes and Recovery Services fail for various reasons: configuration errors, missing dependencies, resource limits, or crashes. Knowing how to identify and recover is crucial. ### Common Failure States - inactive (dead) : Service is not running and not scheduled to run. - failed : Service attempted to start but crashed or exited with an error. - activating : Service is in the process of starting. - deactivating : Service is in the process of stopping. - auto-restart : Service will be restarted automatically by systemd. Check for failed units system-wide: systemctl --failed Expected output (if any): UNIT LOAD ACTIVE SUB DESCRIPTION ● myapp.service loaded failed failed My Custom Application If there are failures, investigate with systemctl status myapp.service and journalctl -u myapp.service -b -e (the -e jumps to the end of the log). ### Debugging a Failed Service Suppose a service myapp.service fails. Perform the following: - Check status: systemctl status myapp.service Look for the Active line and any error messages. - View recent logs: journalctl -u myapp.service -n 50 --no-pager This shows the last 50 log lines. - Examine the unit file: systemctl cat myapp.service Check the ExecStart path and command syntax. - Run the ExecStart command manually (if safe) to see errors directly. - Check file permissions and paths. Ensure the executable and configuration files are accessible to the user specified in the unit ( User= directive). - Use systemd-analyze verify to check the unit file syntax: sudo systemd-analyze verify myapp.service ### Recovery Actions Restart a failed service: sudo systemctl restart myapp.service If the service has a dependency issue, restart dependencies as well: sudo systemctl restart myapp.service --no-block Reset failed state: sudo systemctl reset-failed myapp.service Disable a service temporarily: sudo systemctl stop myapp.service sudo systemctl disable myapp.service Mask a service to prevent it from starting: sudo systemctl mask myapp.service To unmask: sudo systemctl unmask myapp.service Rollback an override: If a recent override caused the failure, delete the drop-in file: sudo rm /etc/systemd/system/myapp.service.d/override.conf sudo systemctl daemon-reload sudo systemctl restart myapp.service ### Recovering from Boot Failures If the system fails to boot due to a bad service, use recovery mode or a live CD. Add systemd.unit=rescue.target or systemd.unit=emergency.target to the kernel command line to boot into a minimal environment. Then disable or fix the offending service. ## Operations Checklist Use this checklist to safely manage systemd services in production. ### Before Any Change - [ ] Confirm the exact service name: systemctl list-units --type=service | grep - [ ] Record current state: systemctl status --no-pager > /tmp/service_status_before.txt - [ ] Check dependencies: systemctl list-dependencies - [ ] Ensure you have a rollback plan (e.g., backup of unit file overrides or documented original state). - [ ] Notify affected users or team if the service is critical. ### During Change - [ ] Use sudo systemctl rather than editing files directly when possible. - [ ] For unit file edits, use sudo systemctl edit to create overrides. - [ ] After editing, run sudo systemd-analyze verify to catch syntax errors. - [ ] Reload daemon if unit files changed: sudo systemctl daemon-reload . - [ ] Apply the change: sudo systemctl restart or sudo systemctl reload if supported. ### After Change - [ ] Verify service is active and running: systemctl is-active should return active . - [ ] Check for errors in logs: journalctl -u -n 20 --no-pager . - [ ] Confirm the service is enabled if it should start at boot: systemctl is-enabled . - [ ] Monitor for a few minutes to ensure no unexpected restarts: systemctl status shows the uptime. - [ ] Record the change and any new observations for future reference. Example: Restarting Nginx # Before systemctl status nginx --no-pager | tee /tmp/nginx_status_before.txt # During sudo systemctl restart nginx # After systemctl is-active nginx # expect: active journalctl -u nginx -n 20 --no-pager ## Conclusion Mastering systemd basic commands is essential for anyone managing Linux systems. This guide covered inventory, safe configuration, verification, failure recovery, and an operations checklist. By following the principles of observation, minimal intervention, and verification, you can operate systemd-managed services with confidence. As a next step, practice these commands on a test system. Create a simple service, make changes using overrides, induce failures, and recover. The more familiar you are with systemctl, journalctl, and systemd-analyze, the faster you can diagnose and fix issues in production. Remember to always have a rollback plan and document your actions.