systemd common errors and fixes with practical examples: practical implementation guide

Intro

Systemd is powerful, but its error messages can feel opaque when a unit fails to start or restarts in a loop. This guide focuses on the most common systemd errors you will see in practice, why they happen, how to diagnose them quickly, and safe ways to fix them.

You will get:

• A repeatable troubleshooting workflow
• A quick reference of inspection commands
• Practical errors, root causes, and copy-paste fixes
• A local pilot plan to practice safely before changing production

Troubleshooting Workflow Overview

Use this sequence end-to-end. It is fast, safe, and repeatable.

1. Identify the unit and reproduce safely

• Know the exact unit name (e.g., nginx.service). Avoid guessing.
• If possible, reproduce in a non-production environment first.

1. Inspect status first

systemctl status nginx.service --no-pager -l

Look for: Active state, ExecStart, Main PID, code/status, recent log lines.

1. Read the logs with context

journalctl -xeu nginx.service --no-pager
# Or restrict to current boot
journalctl -u nginx.service -b --no-pager

1. Verify unit syntax and see what file systemd loads

systemctl cat nginx.service
systemctl show nginx.service -p FragmentPath -p DropInPaths
systemd-analyze verify /etc/systemd/system/nginx.service

1. Test the command outside systemd

• Run the ExecStart command manually as the service user if applicable.
• Check permissions and existence of binaries and scripts.
• For scripts, confirm executable bit and correct shebang.

1. Check ordering and dependencies

systemctl list-dependencies --reverse nginx.service
systemd-analyze critical-chain nginx.service

Adjust Wants=, Requires=, and After= as needed.

1. Review sandboxing and filesystems

• Options like ProtectSystem, ReadWritePaths, PrivateTmp, DynamicUser, and WorkingDirectory can block writes or access.
• If writes fail, either relax protections or write to allowed paths like RuntimeDirectory.

1. Apply changes safely and reload

systemctl daemon-reload
systemctl restart nginx.service

Prefer using drop-ins via:

systemctl edit nginx.service

This creates /etc/systemd/system/nginx.service.d/override.conf.

1. Confirm and monitor

systemctl is-active nginx.service && echo OK
journalctl -u nginx.service -b -n 50 --no-pager

Diagnostics Quick Reference

• Status and last logs

systemctl status <unit> --no-pager -l
journalctl -xeu <unit> --no-pager

• Show loaded unit and drop-ins

systemctl cat <unit>
systemctl show <unit> -p FragmentPath -p DropInPaths

• Validate unit syntax

systemd-analyze verify /etc/systemd/system/<unit>

• Boot-scoped logs

journalctl -u <unit> -b --no-pager

• Dependencies and critical chain

systemctl list-dependencies --reverse <unit>
systemd-analyze critical-chain <unit>

• Environment

systemctl show <unit> -p Environment -p EnvironmentFiles

Common Errors and Fixes

Below are the errors you will likely see, with causes, diagnostics, and safe fixes.

1) "Unit <name>.service not found"

Symptom:

systemctl start myapp
Failed to start myapp.service: Unit myapp.service not found.

Why it happens:

• The unit file is missing or named incorrectly.
• You created or edited a unit but did not run daemon-reload.

Diagnose:

ls /etc/systemd/system/myapp.service /lib/systemd/system/myapp.service
systemctl daemon-reload && systemctl list-unit-files | grep myapp

Fix:

• Ensure the file is placed at /etc/systemd/system/myapp.service.
• Correct the unit name and extension.
• Reload and start:

systemctl daemon-reload
systemctl enable --now myapp.service

2) "Unit is masked"

Symptom:

systemctl start docker
Failed to start docker.service: Unit docker.service is masked.

Why it happens:

• The unit is linked to /dev/null to prevent activation.

Diagnose:

ls -l /etc/systemd/system/docker.service
systemctl status docker

Fix (understand why it was masked first):

systemctl unmask docker.service
systemctl enable --now docker.service

3) "Failed at step EXEC spawning ...: No such file or directory"

Symptom:

systemd[1]: myapp.service: Failed at step EXEC spawning /usr/local/bin/myapp: No such file or directory

Why it happens:

• ExecStart path is wrong or missing.
• Script has a bad shebang interpreter path.

Diagnose:

file /usr/local/bin/myapp
head -n1 /usr/local/bin/myapp
which myapp

Fix:

• Point ExecStart to the correct absolute path.
• For scripts, use a valid shebang like:

#!/usr/bin/env bash

• Ensure executable bit:

chmod +x /usr/local/bin/myapp

4) "Exec format error" or permission denied for scripts

Symptom:

... myapp.sh: Exec format error

Why it happens:

• Windows CRLF line endings.
• Not executable or missing interpreter.

Diagnose and fix:

file myapp.sh                # shows CRLF if present
sed -i 's/\r$//' myapp.sh    # convert to LF
chmod +x myapp.sh
head -n1 myapp.sh            # confirm shebang

5) "Start request repeated too quickly" (crash loop)

Symptom:

systemd[1]: myapp.service: Start request repeated too quickly.

Why it happens:

• The service exits rapidly, often with Restart=always.

Diagnose:

journalctl -u myapp -b -n 100 --no-pager
systemctl show myapp -p Restart -p StartLimitBurst -p StartLimitIntervalSec

Fix:

• Fix the underlying crash first (wrong config, missing env, bad path).
• Optionally ease limits while debugging:

[Unit]
StartLimitIntervalSec=60
StartLimitBurst=10

[Service]
Restart=on-failure
RestartSec=2s

Apply via:

systemctl edit myapp.service
systemctl daemon-reload && systemctl restart myapp

6) "Dependency failed" or ordering issues

Symptom:

systemd[1]: Dependency failed for myapp.service

Why it happens:

• The unit requires another unit that is inactive.
• Network is not ready but the service expects it.

Diagnose:

systemctl list-dependencies --reverse myapp.service
systemd-analyze critical-chain myapp.service

Fix (example: wait for network):

[Unit]
Wants=network-online.target
After=network-online.target

On systems using systemd-networkd, also enable:

systemctl enable --now systemd-networkd-wait-online.service

7) EnvironmentFile not found

Symptom:

... Failed to load environment files: No such file or directory

Why it happens:

• EnvironmentFile path is wrong or missing.

Diagnose:

systemctl show myapp -p EnvironmentFiles
ls -l /etc/default/myapp.env

Fix:

• Create the file and secure permissions:

install -m 0640 -o root -g root /dev/null /etc/default/myapp.env
echo 'MYAPP_PORT=8080' | tee -a /etc/default/myapp.env >/dev/null

• In the unit:

[Service]
EnvironmentFile=/etc/default/myapp.env

• Reload and restart.

8) WorkingDirectory does not exist

Symptom:

... myapp.service: WorkingDirectory=/opt/myapp: No such file or directory

Why it happens:

• Directory is missing or owned by the wrong user.

Fix:

install -d -o myuser -g myuser /opt/myapp

Or use RuntimeDirectory for ephemeral paths:

[Service]
RuntimeDirectory=myapp
WorkingDirectory=/run/myapp

9) Permission denied due to hardening options

Symptom:

... myapp: permission denied writing /var/log/myapp/output.log

Why it happens:

• ProtectSystem=strict or similar prevents writes.

Diagnose:

systemctl cat myapp | sed -n '/\[Service\]/,$p'

Fix safely:

• Prefer writing to a RuntimeDirectory and let systemd manage it:

[Service]
RuntimeDirectory=myapp
# app writes to /run/myapp/

• Or allow specific write paths:

[Service]
ProtectSystem=strict
ReadWritePaths=/var/log/myapp

10) Invalid unit fields or typos

Symptom:

... Failed to parse service type, ignoring: simple2

Why it happens:

• Typos in fields like Type, ExecStart, or section names.

Fix:

• Compare against systemd.unit and systemd.service docs.
• Validate:

systemd-analyze verify /etc/systemd/system/myapp.service

11) "Failed to connect to bus: No such file or directory" in containers

Symptom:

systemctl status
Failed to connect to bus: No such file or directory

Why it happens:

• Most containers do not run systemd as PID 1.

Fix and guidance:

• Do not rely on systemctl inside simple containers. Run your process in the foreground and manage it with the container runtime.
• If you truly need systemd in a container, use a base image designed for it and start the container with appropriate init settings. Evaluate security and complexity before proceeding.

Local Pilot Plan

Start small, measurable, and easy to inspect locally.

Goal: Bring up a minimal service, trigger two common errors, and fix them.

1. Create a simple service

cat >/usr/local/bin/hello.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
: "${HELLO_MSG:=Hello}"
echo "$(date -Is) $HELLO_MSG" | tee -a /var/log/hello/hello.log
sleep 2
EOF
chmod +x /usr/local/bin/hello.sh
install -d -m 0755 /var/log/hello

cat >/etc/systemd/system/hello.service <<'EOF'
[Unit]
Description=Hello demo service
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
EnvironmentFile=-/etc/default/hello
ExecStart=/usr/local/bin/hello.sh
Restart=on-failure
RestartSec=1s
# Harden while allowing log writes
ProtectSystem=strict
ReadWritePaths=/var/log/hello

[Install]
WantedBy=multi-user.target
EOF

install -m 0644 /dev/null /etc/default/hello
printf 'HELLO_MSG=Hello from systemd\n' >> /etc/default/hello

systemctl daemon-reload
systemctl enable --now hello.service
systemctl status hello --no-pager -l
journalctl -u hello -b -n 50 --no-pager

1. Pilot check: validate behavior

• Expect the service to start and write to /var/log/hello/hello.log.
• Confirm with:

tail -n 5 /var/log/hello/hello.log

1. Introduce Error A: bad ExecStart path

systemctl stop hello
systemctl edit hello.service

Add in the drop-in:

[Service]
ExecStart=
ExecStart=/usr/local/bin/hello-typo.sh

Then:

systemctl daemon-reload
systemctl start hello
systemctl status hello -l --no-pager

Observe: Failed at step EXEC spawning ... No such file or directory.

Fix A:

systemctl edit hello.service

Replace with:

[Service]
ExecStart=
ExecStart=/usr/local/bin/hello.sh

Reload and restart.

1. Introduce Error B: crash loop

sed -i '1i exit 1' /usr/local/bin/hello.sh   # force failure at top
systemctl restart hello
journalctl -xeu hello --no-pager

Observe: Start request repeated too quickly if it crashes fast.

Fix B:

• Remove the injected exit:

sed -i '1{/exit 1/d;}' /usr/local/bin/hello.sh

• Optionally tune limits while testing:

systemctl edit hello.service

Add:

[Unit]
StartLimitIntervalSec=60
StartLimitBurst=10

Reload and restart.

1. Inspect and clean up

journalctl -u hello -b --no-pager
systemctl disable --now hello
rm -f /etc/systemd/system/hello.service
systemctl daemon-reload

Why this pilot works:

• It is narrow and measurable: start, log write, two induced failures, two fixes.
• It is easy to inspect locally without side effects.

Conclusion

When systemd fails, follow a consistent workflow: inspect status, read logs, verify the unit, test the command manually, check dependencies and hardening, then apply changes safely and reload. Keep a quick-reference of commands handy and practice with a small local pilot so you can recognize patterns fast. With these steps and examples, you can turn confusing messages into clear root causes and safe, confident fixes.