systemd Architecture Explained with Practical Examples

Introduction

systemd is the default init system and service manager on most modern Linux distributions. It initializes the system, manages services, handles logging, and enforces resource control through cgroups. For developers, DevOps engineers, and technical teams, a working knowledge of systemd's architecture is essential for diagnosing issues, tuning service behavior, and ensuring reliable operations.

This article explains systemd's architecture with practical examples. You will learn to identify the systemd version and deployment topology, inspect unit state, configure services safely, diagnose failures, and plan recovery. Each section includes commands, expected output, failure signals, and recovery steps. The goal is operational safety: observe before changing, limit the blast radius, use placeholders for sensitive data, verify the result, and document recovery paths.

Version and Environment Inventory

Before interacting with systemd, establish the version and environment. Knowing the version matters because systemd features and defaults change between releases. For example, systemd 240 introduced RestartKillSignal=, and version 249 changed the default for Restart= to on-failure for services with Restart=always and a timeout.

Gather version information

Run the following commands on the target system:

systemctl --version
uname -r
cat /etc/os-release

Example output:

systemd 249 (249.11-0ubuntu3.6)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
Linux 5.15.0-60-generic
NAME="Ubuntu"
VERSION="22.04.1 LTS (Jammy Jellyfish)"

This output tells you:

• systemd version 249 with specific compile-time features.
• The kernel version (5.15.0) and distribution (Ubuntu 22.04).
• The cgroup hierarchy is unified (cgroup v2), which affects resource control configuration.

Check that the cgroup version is v2, as some older containers or tools may assume v1. To confirm:

stat -fc %T /sys/fs/cgroup/

Output should be cgroup2fs. If it is tmpfs, the system uses legacy or hybrid cgroups.

Deployment topology

Determine how systemd is used in your environment. Is it a physical host, a virtual machine, or a container? In containers, systemd may not be PID 1, and some features are unavailable. Check:

ps -p 1 -o comm=

Expected output is systemd on a host. If it is bash or sh, you are in a container without a full init system.

For service inspection, identify the service name and its unit file:

systemctl cat sshd.service

This prints the unit file and any drop-in overrides. Note the [Unit], [Service], and [Install] sections.

Prerequisites for safe operation

• You have shell access with permission to run systemctl (usually sudo or root).
• The system is not in emergency or rescue mode unless that is the target.
• You understand that changing unit files affects service behavior and may require a reload or restart.

Read-only observation

Before making changes, capture the current state with timestamps:

date --utc
systemctl is-system-running
systemctl list-units --type=service --state=running

The is-system-running command returns running, degraded, maintenance, stopping, or initializing. A degraded state indicates at least one failed unit. Record this output for later comparison.

Safe Configuration Path

systemd configuration changes should be minimal, tested, and reversible. This section walks through adding a custom service, modifying an existing unit via a drop-in, and rolling back safely.

Example: Creating a simple service

Assume you need to run a script at startup as a specific user. First, write the script and make it executable:

sudo mkdir -p /usr/local/lib/myservice
sudo tee /usr/local/lib/myservice/myscript.sh > /dev/null <<'EOF'
#!/bin/bash
while true; do
  echo "Service running at $(date)" >> /var/log/myservice.log
  sleep 60
done
EOF
sudo chmod +x /usr/local/lib/myservice/myscript.sh

Create a unit file at /etc/systemd/system/myservice.service:

[Unit]
Description=My Custom Service
After=network.target

[Service]
Type=simple
User=myserviceuser
ExecStart=/usr/local/lib/myservice/myscript.sh
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

Key directives explained

• Type=simple: systemd considers the service started immediately after forking the process. Suitable for long-running foreground processes.
• User=myserviceuser: Runs the service as a non-root user for security. Create the user first: sudo useradd -r -s /usr/sbin/nologin myserviceuser.
• Restart=on-failure: Restarts only on unclean exit code or signal.
• RestartSec=5s: Wait 5 seconds between restarts.

Applying the configuration safely

After creating the unit file, reload the systemd manager to pick up changes, then start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start myservice.service
sudo systemctl status myservice.service

Expected output (partial):

● myservice.service - My Custom Service
     Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2023-01-02 12:34:56 UTC; 5s ago
   Main PID: 1234 (myscript.sh)
      Tasks: 1 (limit: 2356)
     Memory: 1.0M
        CPU: 20ms
     CGroup: /system.slice/myservice.service
             └─1234 /bin/bash /usr/local/lib/myservice/myscript.sh

Verify the service is running and log output appears:

sudo tail /var/log/myservice.log

To enable it at boot:

sudo systemctl enable myservice.service

Using drop-in overrides

Instead of editing the original unit file, use a drop-in to override specific settings. This preserves the original and makes rollback easy. For example, to change the restart behavior without modifying the unit file:

sudo mkdir -p /etc/systemd/system/myservice.service.d
sudo tee /etc/systemd/system/myservice.service.d/override.conf > /dev/null <<'EOF'
[Service]
Restart=always
RestartSec=10s
EOF
sudo systemctl daemon-reload
sudo systemctl restart myservice.service

Verify the override is applied:

systemctl cat myservice.service | grep -A2 'override'

Output shows both the original and override sections.

Blast radius and recovery

Changing Restart may increase log noise if the service fails repeatedly. Before applying, consider the impact. If the change causes problems, remove the drop-in and reload:

sudo rm /etc/systemd/system/myservice.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart myservice.service

Always test configuration changes in a staging environment first.

Verification and Diagnostics

After configuration changes, verify the service behaves as expected and diagnose any issues. systemd provides several commands for introspection.

Checking service status

systemctl status gives an overview. For a failed service, it shows relevant log lines and exit codes.

systemctl status myservice.service

If the service fails, output includes:

● myservice.service - My Custom Service
     Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2023-01-02 12:40:00 UTC; 10s ago
    Process: 1235 ExecStart=/usr/local/lib/myservice/myscript.sh (code=exited, status=1/FAILURE)
   Main PID: 1235 (code=exited, status=1/FAILURE)

The Result: exit-code and status=1/FAILURE indicate the script exited with code 1. Investigate the script's logic or permissions.

Viewing logs

Use journalctl for detailed logs:

journalctl -u myservice.service -e

-e jumps to the end. To follow logs in real time:

journalctl -u myservice.service -f

Filter by time:

journalctl -u myservice.service --since "2023-01-02 12:30:00" --until "2023-01-02 12:45:00"

Inspecting unit dependencies

Understand the service's dependency tree:

systemctl list-dependencies myservice.service

This shows what the service requires and what requires it. If a dependency fails, the service may not start.

Verifying resource limits

Use cgroup-related commands to check resource usage:

systemctl show myservice.service | grep -E 'MemoryCurrent|CPUUsageNSec|TasksCurrent'

Example output:

MemoryCurrent=1024000
CPUUsageNSec=25000000
TasksCurrent=1

These values help identify memory leaks or high CPU usage.

Testing configuration validity

Before restarting, verify unit file syntax:

systemd-analyze verify myservice.service

No output means no errors. If there are warnings or errors, they are printed and should be addressed.

Failure Modes and Recovery

systemd services can fail for various reasons: incorrect ExecStart path, missing dependencies, permission issues, or resource exhaustion. This section covers common failure patterns and recovery steps.

Failure: ExecStart path not executable

If the executable path is wrong or lacks execute permission, the service fails immediately.

Symptom: status=203/EXEC in systemctl status.

Diagnosis:

systemctl status myservice.service | grep ExecStart
ls -l /usr/local/lib/myservice/myscript.sh

Recovery: Correct the path or permission, then reload and restart:

sudo chmod +x /usr/local/lib/myservice/myscript.sh
sudo systemctl daemon-reload
sudo systemctl restart myservice.service

Failure: Port already in use

If a service binds to a specific port and the port is taken, it exits with an error like bind: address already in use.

Diagnosis:

journalctl -u myservice.service -e
ss -tulpn | grep :8080

Recovery: Stop the conflicting process or change the port in the service configuration.

Failure: Dependency not met

If a service depends on another unit that is inactive, it may not start.

Symptom: Condition: start condition failed or dependency not met in logs.

Diagnosis:

systemctl list-dependencies myservice.service
systemctl status network.target

Recovery: Start the dependency first or adjust the After= and Requires= directives appropriately.

Recovery from a broken unit file

If a unit file has a syntax error, daemon-reload fails, and the service may disappear.

Symptom: systemctl reports Failed to parse or Invalid argument.

Recovery:

journalctl -xe | grep -i 'myservice'

Edit the unit file, fix the syntax, and run systemd-analyze verify again.

Emergency mode recovery

If the system boots into emergency mode, use a rescue shell to inspect. Common causes include failed mount units or fsck issues.

systemctl list-units --failed
journalctl -b -p err

To exit emergency mode after fixing, run systemctl default or reboot.

Common Pitfalls and How to Avoid Them

Many systemd issues stem from misunderstandings or oversights. Here are frequent pitfalls, why they happen, and how to avoid or recover from them.

Editing unit files without daemon-reload

• Why it happens: Users modify a unit file and expect changes to take effect immediately.
• Consequence: The running systemd manager still uses the old configuration.
• How to avoid: Always run sudo systemctl daemon-reload after editing unit files or drop-ins.
• Recovery: Run the reload command and restart the service.

Using Restart=always without a rate limit

• Why it happens: A service that crashes continuously gets restarted endlessly, filling logs and consuming CPU.
• Consequence: System load increases, and logs become unmanageable.
• How to avoid: Use Restart=on-failure and set RestartSec= appropriately. Consider StartLimitBurst= and StartLimitIntervalSec= to cap restart attempts.
• Recovery: Stop the service, fix the root cause, then start it again.

Ignoring environment variables in unit files

• Why it happens: Scripts assume certain environment variables are set, but systemd does not source /etc/profile or user shell profiles.
• Consequence: The service fails or behaves unexpectedly.
• How to avoid: Use Environment= or EnvironmentFile= directives in the unit file.
• Recovery: Add the missing variables and reload.

Not specifying a proper process type

• Why it happens: The default Type=simple may not suit daemons that fork to the background.
• Consequence: systemd may kill the process or report incorrect status.
• How to avoid: Choose the correct Type= (e.g., forking, notify, oneshot) based on how the executable behaves.
• Recovery: Adjust Type= and restart.

Running services as root unnecessarily

• Why it happens: Convenience leads to using the default root user.
• Consequence: Security risk if the service is compromised.
• How to avoid: Create a dedicated system user and set User= and Group= in the unit file.
• Recovery: Change ownership of necessary files and update the unit file.

Operations Checklist

Use this checklist before and after making systemd changes to ensure safety and observability.

Ownership note: Each item has a single accountable owner, typically the service owner or the DevOps engineer on call. The review frequency is defined per item but should be revisited monthly or after any incident.

Conclusion

systemd's architecture, while complex, becomes manageable with a systematic approach to observation, configuration, and recovery. By understanding the core components (unit files, cgroups, journal, and the manager) and using the right commands, you can maintain stable and secure services.

Start by applying one low-risk verification in your environment:

1. Record the current systemd version and system state.
2. Pick a service you own and inspect its unit file with systemctl cat.
3. Verify syntax with systemd-analyze verify.
4. Check dependencies and logs to understand its behavior.
5. If you need to change a setting, use a drop-in override and follow the safe configuration path.

By making failure visible, protecting sensitive values, limiting changes to the intended resource, and defining recovery verification in advance, you build operational confidence. The next time a service misbehaves, you will have the tools to diagnose and fix it quickly.