## Intro Systemd is powerful, but its error messages can feel opaque when a unit fails to start or restarts in a loop. This guide focuses on the most common systemd errors you will see in practice, why they happen, how to diagnose them quickly, and safe ways to fix them. You will get: - A repeatable troubleshooting workflow - A quick reference of inspection commands - Practical errors, root causes, and copy-paste fixes - A local pilot plan to practice safely before changing production ## Troubleshooting Workflow Overview Use this sequence end-to-end. It is fast, safe, and repeatable. - Identify the unit and reproduce safely - Know the exact unit name (e.g., nginx.service ). Avoid guessing. - If possible, reproduce in a non-production environment first. - Inspect status first systemctl status nginx.service --no-pager -l Look for: Active state, ExecStart, Main PID, code/status, recent log lines. - Read the logs with context journalctl -xeu nginx.service --no-pager # Or restrict to current boot journalctl -u nginx.service -b --no-pager - Verify unit syntax and see what file systemd loads systemctl cat nginx.service systemctl show nginx.service -p FragmentPath -p DropInPaths systemd-analyze verify /etc/systemd/system/nginx.service - Test the command outside systemd - Run the ExecStart command manually as the service user if applicable. - Example: sudo -u nginx /usr/sbin/nginx -t to test nginx configuration. - Check permissions and existence of binaries and scripts. - For scripts, confirm executable bit and correct shebang. - Check ordering and dependencies systemctl list-dependencies --reverse nginx.service systemd-analyze critical-chain nginx.service Adjust Wants= , Requires= , and After= as needed. The critical-chain output shows which units delay startup. - Review sandboxing and filesystems - Options like ProtectSystem , ReadWritePaths , PrivateTmp , DynamicUser , and WorkingDirectory can block writes or access. - If writes fail, either relax protections or write to allowed paths like RuntimeDirectory . - Apply changes safely and reload systemctl daemon-reload systemctl restart nginx.service Prefer using drop-ins via: systemctl edit nginx.service This creates /etc/systemd/system/nginx.service.d/override.conf . - Confirm and monitor systemctl is-active nginx.service && echo OK journalctl -u nginx.service -b -n 50 --no-pager ## Diagnostics Quick Reference - Status and last logs systemctl status --no-pager -l journalctl -xeu --no-pager - Show loaded unit and drop-ins systemctl cat systemctl show -p FragmentPath -p DropInPaths - Validate unit syntax systemd-analyze verify /etc/systemd/system/ - Boot-scoped logs journalctl -u -b --no-pager - Dependencies and critical chain systemctl list-dependencies --reverse systemd-analyze critical-chain - Environment systemctl show -p Environment -p EnvironmentFiles ## Common Errors and Fixes Below are the errors you will likely see, with causes, diagnostics, and safe fixes. ### 1) "Unit .service not found" Symptom: systemctl start myapp Failed to start myapp.service: Unit myapp.service not found. Why it happens: - The unit file is missing or named incorrectly. - You created or edited a unit but did not run daemon-reload . Diagnose: ls /etc/systemd/system/myapp.service /lib/systemd/system/myapp.service systemctl daemon-reload && systemctl list-unit-files | grep myapp Fix: - Ensure the file is placed at /etc/systemd/system/myapp.service . - Correct the unit name and extension. - Reload and start: systemctl daemon-reload systemctl enable --now myapp.service ### 2) "Unit is masked" Symptom: systemctl start docker Failed to start docker.service: Unit docker.service is masked. Why it happens: - The unit is linked to /dev/null to prevent activation. Diagnose: ls -l /etc/systemd/system/docker.service systemctl status docker Fix (understand why it was masked first): systemctl unmask docker.service systemctl enable --now docker.service ### 3) "Failed at step EXEC spawning ...: No such file or directory" Symptom: systemd[1]: myapp.service: Failed at step EXEC spawning /usr/local/bin/myapp: No such file or directory Why it happens: - ExecStart path is wrong or missing. - Script has a bad shebang interpreter path. Diagnose: file /usr/local/bin/myapp head -n1 /usr/local/bin/myapp which myapp Fix: - Point ExecStart to the correct absolute path. - For scripts, use a valid shebang like: #!/usr/bin/env bash - Ensure executable bit: chmod +x /usr/local/bin/myapp ### 4) "Exec format error" or permission denied for scripts Symptom: ... myapp.sh: Exec format error Why it happens: - Windows CRLF line endings. - Not executable or missing interpreter. Diagnose and fix: file myapp.sh # shows CRLF if present sed -i 's/ $//' myapp.sh # convert to LF chmod +x myapp.sh head -n1 myapp.sh # confirm shebang ### 5) "Start request repeated too quickly" (crash loop) Symptom: systemd[1]: myapp.service: Start request repeated too quickly. Why it happens: - The service exits rapidly, often with Restart=always . Diagnose: journalctl -u myapp -b -n 100 --no-pager systemctl show myapp -p Restart -p StartLimitBurst -p StartLimitIntervalSec Fix: - Fix the underlying crash first (wrong config, missing env, bad path). - Optionally ease limits while debugging: [Unit] StartLimitIntervalSec=60 StartLimitBurst=10 [Service] Restart=on-failure RestartSec=2s Apply via: systemctl edit myapp.service systemctl daemon-reload && systemctl restart myapp ### 6) "Dependency failed" or ordering issues Symptom: systemd[1]: Dependency failed for myapp.service Why it happens: - The unit requires another unit that is inactive. - Network is not ready but the service expects it. Diagnose: systemctl list-dependencies --reverse myapp.service systemd-analyze critical-chain myapp.service Fix (example: wait for network): [Unit] Wants=network-online.target After=network-online.target On systems using systemd-networkd, also enable: systemctl enable --now systemd-networkd-wait-online.service ### 7) EnvironmentFile not found Symptom: ... Failed to load environment files: No such file or directory Why it happens: - EnvironmentFile path is wrong or missing. Diagnose: systemctl show myapp -p EnvironmentFiles ls -l /etc/default/myapp.env Fix: - Create the file and secure permissions: install -m 0640 -o root -g root /dev/null /etc/default/myapp.env echo 'MYAPP_PORT=8080' | tee -a /etc/default/myapp.env >/dev/null - In the unit: [Service] EnvironmentFile=/etc/default/myapp.env - Reload and restart. ### 8) WorkingDirectory does not exist Symptom: ... myapp.service: WorkingDirectory=/opt/myapp: No such file or directory Why it happens: - Directory is missing or owned by the wrong user. Fix: install -d -o myuser -g myuser /opt/myapp Or use RuntimeDirectory for ephemeral paths: [Service] RuntimeDirectory=myapp WorkingDirectory=/run/myapp ### 9) Permission denied due to hardening options Symptom: ... myapp: permission denied writing /var/log/myapp/output.log Why it happens: - ProtectSystem=strict or similar prevents writes. Diagnose: systemctl cat myapp | sed -n '/\[Service\]/,$p' Fix safely: - Prefer writing to a RuntimeDirectory and let systemd manage it: [Service] RuntimeDirectory=myapp # app writes to /run/myapp/ - Or allow specific write paths: [Service] ProtectSystem=strict ReadWritePaths=/var/log/myapp ### 10) Invalid unit fields or typos Symptom: ... Failed to parse service type, ignoring: simple2 Why it happens: - Typos in fields like Type, ExecStart, or section names. Fix: - Compare against systemd.unit and systemd.service docs. - Validate: systemd-analyze verify /etc/systemd/system/myapp.service ### 11) "Failed to connect to bus: No such file or directory" in containers Symptom: systemctl status Failed to connect to bus: No such file or directory Why it happens: - Most containers do not run systemd as PID 1. Fix and guidance: - Do not rely on systemctl inside simple containers. Run your process in the foreground and manage it with the container runtime. - If you truly need systemd in a container, use a base image designed for it and start the container with appropriate init settings. Evaluate security and complexity before proceeding. ## Local Pilot Plan Start small, measurable, and easy to inspect locally. Goal: Bring up a minimal service, trigger two common errors, and fix them. - Create a simple service cat >/usr/local/bin/hello.sh <<'EOF' #!/usr/bin/env bash set -euo pipefail : "${HELLO_MSG:=Hello}" echo "$(date -Is) $HELLO_MSG" | tee -a /var/log/hello/hello.log sleep 2 EOF chmod +x /usr/local/bin/hello.sh install -d -m 0755 /var/log/hello cat >/etc/systemd/system/hello.service <<'EOF' [Unit] Description=Hello demo service Wants=network-online.target After=network-online.target [Service] Type=simple EnvironmentFile=-/etc/default/hello ExecStart=/usr/local/bin/hello.sh Restart=on-failure RestartSec=1s # Harden while allowing log writes ProtectSystem=strict ReadWritePaths=/var/log/hello [Install] WantedBy=multi-user.target EOF install -m 0644 /dev/null /etc/default/hello printf 'HELLO_MSG=Hello from systemd\n' >> /etc/default/hello systemctl daemon-reload systemctl enable --now hello.service systemctl status hello --no-pager -l journalctl -u hello -b -n 50 --no-pager - Pilot check: validate behavior - Expect the service to start and write to /var/log/hello/hello.log . - Confirm with: tail -n 5 /var/log/hello/hello.log - Introduce Error A: bad ExecStart path systemctl stop hello systemctl edit hello.service Add in the drop-in: [Service] ExecStart= ExecStart=/usr/local/bin/hello-typo.sh Then: systemctl daemon-reload systemctl start hello systemctl status hello -l --no-pager Observe: Failed at step EXEC spawning ... No such file or directory . Fix A: systemctl edit hello.service Replace with: [Service] ExecStart= ExecStart=/usr/local/bin/hello.sh Reload and restart. - Introduce Error B: crash loop sed -i '1i exit 1' /usr/local/bin/hello.sh # force failure at top systemctl restart hello journalctl -xeu hello --no-pager Observe: Start request repeated too quickly if it crashes fast. Fix B: - Remove the injected exit: sed -i '1d' /usr/local/bin/hello.sh - Optionally tune limits while testing: systemctl edit hello.service Add: [Unit] StartLimitIntervalSec=60 StartLimitBurst=10 Reload and restart. - Inspect and clean up journalctl -u hello -b --no-pager systemctl disable --now hello rm -f /etc/systemd/system/hello.service systemctl daemon-reload Why this pilot works: - It is narrow and measurable: start, log write, two induced failures, two fixes. - It is easy to inspect locally without side effects. ## Conclusion When systemd fails, follow a consistent workflow: inspect status, read logs, verify the unit, test the command manually, check dependencies and hardening, then apply changes safely and reload. Keep a quick-reference of commands handy and practice with a small local pilot so you can recognize patterns fast. With these steps and examples, you can turn confusing messages into clear root causes and safe, confident fixes.