systemd Troubleshooting with Practical Examples: A Hands-On Guide

Learn to diagnose and fix common systemd failures with practical commands, log analysis, and recovery workflows. This guide covers service failures, dependency issues, resource limits, and safe rollback procedures for developers and DevOps teams.

Intro

systemd is the default init system and service manager on most modern Linux distributions. It starts, stops, and supervises services, manages sockets, devices, mounts, and timers, and centralizes logging via the journal. While systemd is powerful, its complexity means failures can be confusing. A service may fail to start, hang on shutdown, consume too much memory, or misbehave due to incorrect dependencies. Troubleshooting systemd requires a systematic approach: checking status, reading logs, understanding unit files, and applying safe configuration changes.

This guide provides practical, hands-on examples for diagnosing and fixing common systemd issues. We'll cover essential commands, log analysis, safe overrides using drop-in files, verification steps, and recovery procedures. Whether you're a developer running a local service or a DevOps consultant managing production fleets, these techniques will help you resolve problems quickly and prevent recurrence. By the end, you'll have a checklist for routine systemd operations and confidence to handle failures when they arise.

Version and Environment Inventory

Before troubleshooting, establish your environment. Knowing the systemd version and the topology of your services helps you understand available features and potential compatibility issues. Always document the OS distribution, kernel version, systemd version, and critical service list. This makes it easier to search for known bugs and to reproduce issues on a test system.

Check systemd version

Run:

systemctl --version

Example output:

systemd 249 (249.11-0ubuntu3.6)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +UDEV +LZMA

The version number and compile-time features help you know which directives are available. For example, MemoryMax was introduced in systemd 232, so if you are on an older version, you may need to use MemoryLimit instead.

List all units and their states

systemctl list-units --type=service --all

This shows every service unit, regardless of whether it is loaded, active, inactive, or failed. The exit code of this command is not useful for scripting because it returns 0 even if units are failed; instead, use systemctl --failed to list only failed units.

Check boot time and system uptime

systemd-analyze

Output shows the time spent in the kernel before systemd starts, the time spent in the initrd, and the time to reach the graphical target. For example:

Startup finished in 1.234s (kernel) + 3.456s (initrd) + 7.890s (userspace) = 12.58s

If userspace time is high, use systemd-analyze blame to find which services slow the boot.

Identify critical services

List the units that are required for your application to work. For a typical web application, these might include network.target, postgresql.service, nginx.service, and php-fpm.service. Check each with systemctl status <unit> and note their activation state.

Prerequisites for effective troubleshooting:

• Root or sudo access.
• Basic understanding of Linux processes and filesystems.
• Familiarity with your service's intended behavior.

Keep a small text file or wiki page with your environment inventory. Include:

• OS distribution and version: cat /etc/os-release
• Kernel version: uname -r
• systemd version: systemctl --version | head -1
• List of critical services and their expected states.

This information helps you compare with a known good system and speeds up problem identification.

Safe Configuration Path

Avoid editing unit files directly in /lib/systemd/system/ or /usr/lib/systemd/system/ because package updates can overwrite them. Instead, use systemd's drop-in mechanism or copy the unit to /etc/systemd/system/.

The standard search path for unit files is:

• /etc/systemd/system/ (highest priority, admin overrides)
• /run/systemd/system/ (runtime units)
• /usr/lib/systemd/system/ (vendor units)

When you create a file in /etc/systemd/system/ with the same name as a vendor unit, it completely masks the vendor unit. This is sometimes useful, but for small changes, drop-ins are safer because they preserve the original and are easier to remove.

Drop-in overrides

Create a directory named <unit>.d/ and place .conf files inside. For example, to increase the startup timeout for a service called myservice.service:

sudo mkdir -p /etc/systemd/system/myservice.service.d
sudo tee /etc/systemd/system/myservice.service.d/override.conf <<'EOF'
[Service]
TimeoutStartSec=60
EOF
sudo systemctl daemon-reload
sudo systemctl restart myservice

The TimeoutStartSec directive sets the maximum time for the service to start. The default is 90 seconds for most services, but some distributions set a lower default.

Verify the effective configuration

Run:

systemctl cat myservice.service

The output shows the original unit followed by all drop-in files, so you can see exactly what systemd will use. It marks drop-in snippets with a comment line.

Use systemctl edit for convenience

Instead of creating the directory and file manually, you can run:

sudo systemctl edit myservice.service

This opens an editor with a temporary file and creates the drop-in automatically when you save. Use sudo systemctl edit --full myservice.service to edit the entire unit by copying it to /etc/systemd/system/ first.

Common safe overrides

Here are typical changes you might make via drop-in:

• Adjust restart policy: Override Restart=, RestartSec=.
• Set environment variables: Add an [Service] section with Environment="VAR=value".
• Add resource limits: Use MemoryMax, CPUQuota, TasksMax.
• Modify ExecStart: Rarely; instead, consider creating a new unit or masking the old one.

Example drop-in that adds a restart policy and a memory limit:

[Service]
Restart=on-failure
RestartSec=5s
MemoryMax=512M

Place this in /etc/systemd/system/myservice.service.d/limits.conf.

Rollback

If a change causes problems, remove the drop-in directory and reload:

sudo rm -rf /etc/systemd/system/myservice.service.d
sudo systemctl daemon-reload
sudo systemctl restart myservice

Always test changes on a non-production system first. Use systemctl show to inspect current settings after applying an override.

Verification and Diagnostics

Diagnosing systemd issues involves examining service status, logs, and dependencies. This section covers the essential commands and how to interpret their output.

Check service status

Run:

systemctl status myservice.service

Example output:

● myservice.service - Example Web Service
   Loaded: loaded (/etc/systemd/system/myservice.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2024-03-21 14:30:22 UTC; 5min ago
  Process: 1234 ExecStart=/usr/sbin/myservice --config /etc/myservice.conf (code=exited, status=1/FAILURE)
 Main PID: 1234 (code=exited, status=1/FAILURE)

Mar 21 14:30:22 hostname myservice[1234]: Failed to bind to port 8080: Address already in use

The Active line shows the state and the reason. The Process: line gives the exit status. The log lines at the bottom often contain the exact error.

Read logs with journalctl

The journalctl command is the primary tool for viewing logs generated by systemd and services.

Common options:

• -u myservice.service: filter by unit.
• -b: show logs from the current boot.
• --since "10 minutes ago": show recent entries.
• -f: follow live logs (like tail -f).
• -p err: show only error and higher priority.
• --no-pager: disable paging for scripting.

Example:

journalctl -u myservice.service -b --no-pager

To see logs from the previous boot:

journalctl -u myservice.service -b -1

Journal logs persist across reboots only if the journal is configured for persistent storage. Check with journalctl --disk-usage. If it returns 0, logs are volatile and lost on reboot. You can enable persistent storage by creating /var/log/journal:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

Example log extraction and diagnosis

Log line:

Apr 12 14:30:22 hostname myservice[1234]: Failed to bind to port 8080: Address already in use

This indicates another process is already listening on port 8080. Identify the process:

sudo ss -lptn 'sport = :8080'

Output:

State   Recv-Q  Send-Q   Local Address:Port   Peer Address:Port  Process
LISTEN  0       128      0.0.0.0:8080          0.0.0.0:*          users:(("nginx",pid=1235,fd=6))

The fix is to stop the conflicting process or change the port in the service configuration.

Inspect dependencies and ordering

systemctl list-dependencies myservice.service

This shows a tree of units that are required or wanted by the service. Add --reverse to see which units depend on the service:

systemctl list-dependencies --reverse myservice.service

Check for failed units

systemctl --failed

Output lists failed units with a short description. This is a quick health check.

Analyze boot performance

systemd-analyze blame

This lists services by the time they took to start. The first few are often candidates for optimization. Example output:

6.512s postgresql.service
3.204s networkd-dispatcher.service
1.910s dev-sda1.device
1.504s snapd.service
...

Use this to identify services that are slowing boot. You can defer non-essential services or adjust their dependencies.

Verify resource usage

systemctl show myservice.service -p MemoryCurrent -p CPUUsageNSec

Output:

MemoryCurrent=1843200
CPUUsageNSec=5000000000

MemoryCurrent is in bytes. To compare with MemoryMax, also check MemoryMax:

systemctl show myservice.service -p MemoryMax

Other useful properties: TasksCurrent, NRestarts, ExecMainStatus.

Use systemctl is-active and is-enabled

systemctl is-active myservice.service
systemctl is-enabled myservice.service

These return exit codes suitable for scripts: 0 for active/enabled, non-zero otherwise.

For complex services, you can validate the unit file syntax before restarting:

systemd-analyze verify /etc/systemd/system/myservice.service

This catches many configuration errors without affecting the running system.

Failure Modes and Recovery

Understanding common failure patterns helps you react quickly. This section details typical scenarios, their symptoms, diagnosis, and recovery steps.

1. Service fails to start due to missing file or permission

Log line:

Failed to execute command: No such file or directory

Diagnosis: The ExecStart path is wrong or the executable file does not exist. Check the unit file:

systemctl cat myservice.service | grep ExecStart

Verify the file exists and is executable:

ls -l /usr/sbin/myservice
file /usr/sbin/myservice

Fix by correcting the path in the unit file or creating a symlink. If the file exists but permissions are wrong:

sudo chmod 755 /usr/sbin/myservice

If the service needs to run a command as a non-root user and fails due to permissions, check the User= and Group= directives in the unit file.

2. Dependency not met

Log line:

Dependency failed for myservice.service

Diagnosis: A unit that myservice requires has failed. List dependencies:

systemctl list-dependencies myservice.service

Look for failed dependencies and check their status:

systemctl status <failed-dependency>

Resolve the root cause of the dependency failure. For example, if network-online.target fails, ensure the network service is running correctly. After fixing, start the dependency first, then the service.

3. Port already in use

Log line:

bind: Address already in use

Diagnosis: Another process is listening on the same port. Identify the process:

sudo ss -lptn 'sport = :8080'

If the conflicting process is a systemd service, stop and disable it or reconfigure to use a different port. If it is a manual process, terminate it or change your service's port. Always update firewall rules accordingly.

4. Timeout on start

Log line:

Job for myservice.service failed because a timeout was exceeded

Diagnosis: The service took longer than TimeoutStartSec to start. Possible causes: slow network calls, large data initialization, or a deadlock. Investigate by increasing the timeout temporarily and observing logs:

sudo systemctl edit myservice.service

Add:

[Service]
TimeoutStartSec=300

Then reload and restart. If the service starts within the longer timeout, the root cause may be a slow dependency. Optimize the startup process or adjust the timeout permanently.

5. Resource limits exceeded (OOM kill)

Journal shows:

Killed process 1234 (myservice) total-vm:1234567kB, anon-rss:345678kB, file-rss:0kB, shmem-rss:0kB

Diagnosis: The service exceeded its memory limit or the system ran out of memory. Check the service's memory usage and limit:

systemctl show myservice.service -p MemoryCurrent -p MemoryMax

If MemoryMax is set, the service is killed when it exceeds that limit. Fix by increasing the limit via drop-in:

[Service]
MemoryMax=1G

If no limit is set, the system OOM killer acted due to overall memory pressure. Use journalctl -b | grep -i oom to see the OOM killer's decision. Reduce memory usage or add more RAM.

6. Configuration syntax error

Run:

systemd-analyze verify /etc/systemd/system/myservice.service

Example error output:

[/etc/systemd/system/myservice.service:8] Unknown lvalue 'ExecStar' in section 'Service'

Fix the typo in the unit file, then run sudo systemctl daemon-reload. Always verify before restarting.

Rollback strategies

• Keep backups of original unit files before editing. If you edit a vendor unit, copy it to /etc/systemd/system/ and keep a backup.
• Use drop-ins so you can simply remove the override to revert.
• For systemd version upgrades, test on a canary host. Create a snapshot of the unit files and compare behavior before rolling out.

Recovery from a broken boot

If the system does not boot normally, you can enter emergency mode or rescue mode from the boot loader (e.g., GRUB). In GRUB, edit the kernel command line by pressing e, add systemd.unit=rescue.target or systemd.unit=emergency.target, and boot. Once in a shell, diagnose:

systemctl --failed
journalctl -b

To disable a problematic service so the system can boot normally:

systemctl disable myservice.service

Then reboot. After fixing, re-enable the service if needed.

Document each failure and fix for future reference

Create post-mortem notes in a shared wiki or Git repository. Include:

• Symptom and error message.
• Commands used for diagnosis.
• Root cause.
• Fix applied.
• Preventive measures.

This builds institutional knowledge and speeds resolution of similar issues.

Common Pitfalls and How to Avoid Them

Even experienced administrators make mistakes with systemd. This section highlights frequent pitfalls and recovery strategies.

1. Editing vendor unit files directly

Why it happens: It seems quick to modify /usr/lib/systemd/system/myservice.service directly. However, package updates overwrite your changes, causing the problem to reappear or the service to break after upgrade.

How to avoid: Always use drop-in files or copy the unit to /etc/systemd/system/. If you must edit a vendor unit, note it and be prepared to reapply after updates. Use systemctl cat to see all overrides.

2. Forgetting daemon-reload after changes

Why it happens: After editing a unit file or drop-in, systemd does not automatically pick up changes until you run systemctl daemon-reload. Forgetting this step leads to confusion when the service behaves as before.

How to avoid: Always run sudo systemctl daemon-reload after any unit file modification. Consider using systemctl edit which handles this automatically for drop-ins.

3. Misusing Restart=always

Why it happens: Setting Restart=always can lead to a crash loop if the service fails immediately. The service restarts endlessly, flooding logs and consuming resources.

How to avoid: Use Restart=on-failure with a reasonable RestartSec delay. Set StartLimitIntervalSec and StartLimitBurst to control restart attempts. Example drop-in:

[Service]
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3

This means systemd allows at most 3 restarts within 60 seconds, then gives up and marks the unit failed.

4. Ignoring logs and relying on guesswork

Why it happens: Under pressure, admins may skip log analysis and try random fixes. This wastes time and can introduce new issues.

How to avoid: Always start with systemctl status <unit> and journalctl -u <unit>. The error message usually points to the root cause. If logs are unclear, increase verbosity via the service's own configuration or run with debug if supported.

5. Not setting resource limits

Why it happens: Many services have no resource limits by default, so a memory leak can exhaust system memory and trigger the OOM killer, affecting other services.

How to avoid: Set MemoryMax, CPUQuota, and TasksMax in the service unit or drop-in. For example:

[Service]
MemoryMax=512M
CPUQuota=50%
TasksMax=100

Monitor usage with systemctl show and adjust as needed.

6. Overlooking dependency ordering

Why it happens: Services may start before their dependencies are ready because the unit file only declares Wants= but not After=. This causes race conditions.

How to avoid: Explicitly declare ordering with After= and Requires= as appropriate. For network services, use After=network-online.target and Wants=network-online.target to ensure the network is up.

7. Neglecting to test unit changes

Why it happens: Admins may apply changes directly to production without testing, leading to outages.

How to avoid: Use a staging environment or a canary host. Validate syntax with systemd-analyze verify. Use systemctl try-restart to avoid restarting a service that is not running.

Operations Checklist

Use this checklist for routine systemd management and troubleshooting. Assign a single owner to each regular check to ensure accountability. Review frequency is suggested in parentheses.

Regular checks

• [ ] Run systemctl --failed to list failed units. Owner: Operations Engineer (daily).
• [ ] Monitor logs for errors: journalctl -p err -b. Owner: Operations Engineer (daily).
• [ ] Verify critical services are active: systemctl is-active <service> for each. Owner: Service Owner (weekly).
• [ ] Check disk space: df -h and ensure journal isn't too large (journalctl --disk-usage). Owner: System Administrator (weekly).
• [ ] Keep systemd updated: apt upgrade systemd or equivalent. Owner: System Administrator (monthly, following change management).

When a service fails

• [ ] Run systemctl status <service> to get failure message. Owner: On-call engineer (immediate).
• [ ] Check recent logs: journalctl -u <service> -n 50 --no-pager. Owner: On-call engineer (immediate).
• [ ] Inspect dependencies: systemctl list-dependencies <service>. Owner: On-call engineer (immediate).
• [ ] Verify configuration: systemctl cat <service> and systemd-analyze verify. Owner: Service Owner (before restart).
• [ ] Check resource limits: systemctl show <service> -p MemoryMax -p CPUQuota. Owner: Service Owner (before restart).

Before making changes

• [ ] Backup current unit file (if editing directly) or use drop-in. Owner: Engineer making change.
• [ ] Test syntax with systemd-analyze verify. Owner: Engineer making change.
• [ ] Run systemctl daemon-reload after modifications. Owner: Engineer making change.
• [ ] Restart service and observe logs briefly. Owner: Engineer making change.

After changes

• [ ] Confirm systemctl is-active <service> returns active. Owner: Engineer making change.
• [ ] Check for new failures: systemctl --failed. Owner: Engineer making change.
• [ ] Update documentation if configuration changed. Owner: Engineer making change.

Recovery

• [ ] To rollback: remove drop-in or restore original unit file, then daemon-reload and restart. Owner: Engineer making change.
• [ ] If boot issues, use rescue mode to disable problematic service. Owner: Senior System Administrator (immediate, then assign fix).

Review the checklist quarterly to ensure it reflects current infrastructure and ownership changes.

Conclusion

systemd troubleshooting is a systematic process that combines status inspection, log analysis, safe configuration changes, and recovery planning. By following the practices in this guide - maintaining version awareness, using drop-ins for overrides, verifying with systemctl and journalctl, and preparing rollback steps - you can resolve incidents efficiently and prevent future failures.

Key takeaways:

• Always check systemctl status and journalctl first for clues.
• Use drop-in files for configuration changes to preserve original units and ease rollback.
• Verify dependencies, resource limits, and port conflicts for common issues.
• Keep an operations checklist for routine monitoring and quick response.

Remember, every system is different, so adapt these techniques to your environment. Document your findings and share knowledge within your team to build resilience. With practice, you'll turn systemd from a black box into a well-understood component of your infrastructure.