## Introduction systemd is the default init system and service manager on most modern Linux distributions. It initializes the system, manages services, handles logging, and enforces resource control through cgroups. For developers, DevOps engineers, and technical teams, a working knowledge of systemd's architecture is essential for diagnosing issues, tuning service behavior, and ensuring reliable operations. This article explains systemd's architecture with practical examples. You will learn to identify the systemd version and deployment topology, inspect unit state, configure services safely, diagnose failures, and plan recovery. Each section includes commands, expected output, failure signals, and recovery steps. The goal is operational safety: observe before changing, limit the blast radius, use placeholders for sensitive data, verify the result, and document recovery paths. ## Version and Environment Inventory Before interacting with systemd, establish the version and environment. Knowing the version matters because systemd features and defaults change between releases. For example, systemd 240 introduced RestartKillSignal= , and version 249 changed the default for Restart= to on-failure for services with Restart=always and a timeout. ### Gather version information Run the following commands on the target system: systemctl --version uname -r cat /etc/os-release Example output: systemd 249 (249.11-0ubuntu3.6) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified Linux 5.15.0-60-generic NAME="Ubuntu" VERSION="22.04.1 LTS (Jammy Jellyfish)" This output tells you: - systemd version 249 with specific compile-time features. - The kernel version (5.15.0) and distribution (Ubuntu 22.04). - The cgroup hierarchy is unified (cgroup v2), which affects resource control configuration. Check that the cgroup version is v2, as some older containers or tools may assume v1. To confirm: stat -fc %T /sys/fs/cgroup/ Output should be cgroup2fs . If it is tmpfs , the system uses legacy or hybrid cgroups. ### Deployment topology Determine how systemd is used in your environment. Is it a physical host, a virtual machine, or a container? In containers, systemd may not be PID 1, and some features are unavailable. Check: ps -p 1 -o comm= Expected output is systemd on a host. If it is bash or sh , you are in a container without a full init system. For service inspection, identify the service name and its unit file: systemctl cat sshd.service This prints the unit file and any drop-in overrides. Note the [Unit] , [Service] , and [Install] sections. ### Prerequisites for safe operation - You have shell access with permission to run systemctl (usually sudo or root). - The system is not in emergency or rescue mode unless that is the target. - You understand that changing unit files affects service behavior and may require a reload or restart. ### Read-only observation Before making changes, capture the current state with timestamps: date --utc systemctl is-system-running systemctl list-units --type=service --state=running The is-system-running command returns running , degraded , maintenance , stopping , or initializing . A degraded state indicates at least one failed unit. Record this output for later comparison. ## Safe Configuration Path systemd configuration changes should be minimal, tested, and reversible. This section walks through adding a custom service, modifying an existing unit via a drop-in, and rolling back safely. ### Example: Creating a simple service Assume you need to run a script at startup as a specific user. First, write the script and make it executable: sudo mkdir -p /usr/local/lib/myservice sudo tee /usr/local/lib/myservice/myscript.sh > /dev/null <<'EOF' #!/bin/bash while true; do echo "Service running at $(date)" >> /var/log/myservice.log sleep 60 done EOF sudo chmod +x /usr/local/lib/myservice/myscript.sh Create a unit file at /etc/systemd/system/myservice.service : [Unit] Description=My Custom Service After=network.target [Service] Type=simple User=myserviceuser ExecStart=/usr/local/lib/myservice/myscript.sh Restart=on-failure RestartSec=5s [Install] WantedBy=multi-user.target ### Key directives explained - Type=simple : systemd considers the service started immediately after forking the process. Suitable for long-running foreground processes. - User=myserviceuser : Runs the service as a non-root user for security. Create the user first: sudo useradd -r -s /usr/sbin/nologin myserviceuser . - Restart=on-failure : Restarts only on unclean exit code or signal. - RestartSec=5s : Wait 5 seconds between restarts. ### Applying the configuration safely After creating the unit file, reload the systemd manager to pick up changes, then start and enable the service: sudo systemctl daemon-reload sudo systemctl start myservice.service sudo systemctl status myservice.service Expected output (partial): ● myservice.service - My Custom Service Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2023-01-02 12:34:56 UTC; 5s ago Main PID: 1234 (myscript.sh) Tasks: 1 (limit: 2356) Memory: 1.0M CPU: 20ms CGroup: /system.slice/myservice.service └─1234 /bin/bash /usr/local/lib/myservice/myscript.sh Verify the service is running and log output appears: sudo tail /var/log/myservice.log To enable it at boot: sudo systemctl enable myservice.service ### Using drop-in overrides Instead of editing the original unit file, use a drop-in to override specific settings. This preserves the original and makes rollback easy. For example, to change the restart behavior without modifying the unit file: sudo mkdir -p /etc/systemd/system/myservice.service.d sudo tee /etc/systemd/system/myservice.service.d/override.conf > /dev/null <<'EOF' [Service] Restart=always RestartSec=10s EOF sudo systemctl daemon-reload sudo systemctl restart myservice.service Verify the override is applied: systemctl cat myservice.service | grep -A2 'override' Output shows both the original and override sections. ### Blast radius and recovery Changing Restart may increase log noise if the service fails repeatedly. Before applying, consider the impact. If the change causes problems, remove the drop-in and reload: sudo rm /etc/systemd/system/myservice.service.d/override.conf sudo systemctl daemon-reload sudo systemctl restart myservice.service Always test configuration changes in a staging environment first. ## Verification and Diagnostics After configuration changes, verify the service behaves as expected and diagnose any issues. systemd provides several commands for introspection. ### Checking service status systemctl status gives an overview. For a failed service, it shows relevant log lines and exit codes. systemctl status myservice.service If the service fails, output includes: ● myservice.service - My Custom Service Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2023-01-02 12:40:00 UTC; 10s ago Process: 1235 ExecStart=/usr/local/lib/myservice/myscript.sh (code=exited, status=1/FAILURE) Main PID: 1235 (code=exited, status=1/FAILURE) The Result: exit-code and status=1/FAILURE indicate the script exited with code 1. Investigate the script's logic or permissions. ### Viewing logs Use journalctl for detailed logs: journalctl -u myservice.service -e -e jumps to the end. To follow logs in real time: journalctl -u myservice.service -f Filter by time: journalctl -u myservice.service --since "2023-01-02 12:30:00" --until "2023-01-02 12:45:00" ### Inspecting unit dependencies Understand the service's dependency tree: systemctl list-dependencies myservice.service This shows what the service requires and what requires it. If a dependency fails, the service may not start. ### Verifying resource limits Use cgroup-related commands to check resource usage: systemctl show myservice.service | grep -E 'MemoryCurrent|CPUUsageNSec|TasksCurrent' Example output: MemoryCurrent=1024000 CPUUsageNSec=25000000 TasksCurrent=1 These values help identify memory leaks or high CPU usage. ### Testing configuration validity Before restarting, verify unit file syntax: systemd-analyze verify myservice.service No output means no errors. If there are warnings or errors, they are printed and should be addressed. ## Failure Modes and Recovery systemd services can fail for various reasons: incorrect ExecStart path, missing dependencies, permission issues, or resource exhaustion. This section covers common failure patterns and recovery steps. ### Failure: ExecStart path not executable If the executable path is wrong or lacks execute permission, the service fails immediately. Symptom: status=203/EXEC in systemctl status . Diagnosis: systemctl status myservice.service | grep ExecStart ls -l /usr/local/lib/myservice/myscript.sh Recovery: Correct the path or permission, then reload and restart: sudo chmod +x /usr/local/lib/myservice/myscript.sh sudo systemctl daemon-reload sudo systemctl restart myservice.service ### Failure: Port already in use If a service binds to a specific port and the port is taken, it exits with an error like bind: address already in use . Diagnosis: journalctl -u myservice.service -e ss -tulpn | grep :8080 Recovery: Stop the conflicting process or change the port in the service configuration. ### Failure: Dependency not met If a service depends on another unit that is inactive, it may not start. Symptom: Condition: start condition failed or dependency not met in logs. Diagnosis: systemctl list-dependencies myservice.service systemctl status network.target Recovery: Start the dependency first or adjust the After= and Requires= directives appropriately. ### Recovery from a broken unit file If a unit file has a syntax error, daemon-reload fails, and the service may disappear. Symptom: systemctl reports Failed to parse or Invalid argument . Recovery: journalctl -xe | grep -i 'myservice' Edit the unit file, fix the syntax, and run systemd-analyze verify again. ### Emergency mode recovery If the system boots into emergency mode, use a rescue shell to inspect. Common causes include failed mount units or fsck issues. systemctl list-units --failed journalctl -b -p err To exit emergency mode after fixing, run systemctl default or reboot. ## Common Pitfalls and How to Avoid Them Many systemd issues stem from misunderstandings or oversights. Here are frequent pitfalls, why they happen, and how to avoid or recover from them. ### Editing unit files without daemon-reload - Why it happens: Users modify a unit file and expect changes to take effect immediately. - Consequence: The running systemd manager still uses the old configuration. - How to avoid: Always run sudo systemctl daemon-reload after editing unit files or drop-ins. - Recovery: Run the reload command and restart the service. ### Using Restart=always without a rate limit - Why it happens: A service that crashes continuously gets restarted endlessly, filling logs and consuming CPU. - Consequence: System load increases, and logs become unmanageable. - How to avoid: Use Restart=on-failure and set RestartSec= appropriately. Consider StartLimitBurst= and StartLimitIntervalSec= to cap restart attempts. - Recovery: Stop the service, fix the root cause, then start it again. ### Ignoring environment variables in unit files - Why it happens: Scripts assume certain environment variables are set, but systemd does not source /etc/profile or user shell profiles. - Consequence: The service fails or behaves unexpectedly. - How to avoid: Use Environment= or EnvironmentFile= directives in the unit file. - Recovery: Add the missing variables and reload. ### Not specifying a proper process type - Why it happens: The default Type=simple may not suit daemons that fork to the background. - Consequence: systemd may kill the process or report incorrect status. - How to avoid: Choose the correct Type= (e.g., forking , notify , oneshot ) based on how the executable behaves. - Recovery: Adjust Type= and restart. ### Running services as root unnecessarily - Why it happens: Convenience leads to using the default root user. - Consequence: Security risk if the service is compromised. - How to avoid: Create a dedicated system user and set User= and Group= in the unit file. - Recovery: Change ownership of necessary files and update the unit file. ## Operations Checklist Use this checklist before and after making systemd changes to ensure safety and observability.
StepActionCommandExpected ResultOwnerFrequency
1Record current systemd versionsystemctl --versionVersion string matches supported rangeDevOps Engineer (e.g., Priya Shah)Before every change
2Check overall system statesystemctl is-system-runningrunning or degraded (documented)DevOps EngineerBefore every change
3List running services snapshotsystemctl list-units --type=service --state=running > /tmp/services_before.txtFile created with listDevOps EngineerBefore every change
4Make one scoped change(edit unit file or drop-in)Change applied to one service onlyOwner of the service (e.g., Developer Alex Chen)When changing a service
5Reload systemd configurationsudo systemctl daemon-reloadNo error outputDevOps EngineerAfter editing unit files
6Verify unit file syntaxsystemd-analyze verify <unit>No errors or warningsService OwnerAfter change
7Restart/start the servicesudo systemctl restart <unit>Active: active (running)Service OwnerAfter change
8Check service status and logssystemctl status <unit> and journalctl -u <unit> -n 50No failed status; logs show normal operationService OwnerAfter change
9Compare before/after unit listdiff /tmp/services_before.txt <(systemctl list-units --type=service --state=running)Only intended changesDevOps EngineerAfter change
10Document change and rollbackUpdate runbook with rollback stepsRollback tested in stagingService OwnerMonthly audit
Ownership note: Each item has a single accountable owner, typically the service owner or the DevOps engineer on call. The review frequency is defined per item but should be revisited monthly or after any incident. ## Conclusion systemd's architecture, while complex, becomes manageable with a systematic approach to observation, configuration, and recovery. By understanding the core components (unit files, cgroups, journal, and the manager) and using the right commands, you can maintain stable and secure services. Start by applying one low-risk verification in your environment: - Record the current systemd version and system state. - Pick a service you own and inspect its unit file with systemctl cat . - Verify syntax with systemd-analyze verify . - Check dependencies and logs to understand its behavior. - If you need to change a setting, use a drop-in override and follow the safe configuration path. By making failure visible, protecting sensitive values, limiting changes to the intended resource, and defining recovery verification in advance, you build operational confidence. The next time a service misbehaves, you will have the tools to diagnose and fix it quickly.