## Intro Linux configuration is powerful—and unforgiving when something small goes wrong. This guide gives administrators a safe, repeatable way to triage incidents; inspect the effective configuration; validate syntax; make one bounded change; and roll back cleanly. It focuses on real-world categories where mistakes happen: systemd units, network routes and DNS, file ownership and permissions, SSH/sudo access, package or application configuration, storage and mounts, firewall rules, and environment-variable precedence. Use it as a runbook during an outage or as a checklist before deploying changes. ## Safe triage sequence Move methodically. Keep a rollback path at every step. - Capture current state (read-only) - Who and what: id; hostnamectl; uname -a; date -Is - Processes and services: systemctl status my.service; systemctl show -p FragmentPath,Environment my.service; systemctl cat my.service - Logs: journalctl -u my.service --since "-15 min"; journalctl -xeu my.service - Network: ip -br addr; ip route; ss -tulpn - Storage: findmnt -A; cat /etc/fstab - DNS: getent hosts example.org; getent ahosts example.org - Save copies of relevant files with timestamps: cp /etc/app/app.conf /etc/app/app.conf.bak.$(date +%Y%m%d%H%M%S) - Reproduce the symptom - Confirm the error message, failing endpoint, or missing socket. Try the minimal path first (localhost) before remote. - Identify the owning subsystem - Is this service-level (systemd), network, storage, auth (SSH/sudo), application syntax, or firewall? - Inspect the effective configuration - Prefer effective views over guessing: systemctl cat my.service; nginx -T; sshd -T | sort; printenv when running a command interactively. - Compare to intended state - Diff against the last known-good file or a baseline repo copy. Note every delta and its purpose. - Validate syntax (read-only) - systemd-analyze verify /etc/systemd/system/my.service - nginx -t; sshd -t; visudo -c; named-checkconf (BIND); haproxy -c -f /etc/haproxy/haproxy.cfg - Make one bounded change (state-changing, safe) - Apply a minimal, reversible edit (drop-in, small include, or single key). Avoid multiple changes at once. - Reload or restart, then verify - Prefer reload when supported: systemctl reload nginx - Check logs and sockets: journalctl -xeu unit; ss -tulpn | grep :PORT - Keep rollback ready - Restore the .bak and reload if the signal turns red. ## Quick map: symptom to checks to safe action

Symptom	First checks (read-only)	Safe action (state-changing)
Service will not start	systemctl status; journalctl -xeu svc; systemd-analyze verify file	Fix unit file in a drop-in; systemctl daemon-reload; systemctl restart svc
Port not listening	ss -lntp; systemctl status; netstat equivalent	Correct bind address/port; reload; confirm with ss
Can reach localhost but not remote	ip -br addr; ip route get 1.1.1.1; getent hosts name	Add/adjust route; fix DNS; systemctl reload network stack if required
Permission denied on config/file	ls -l; namei -l /path; systemctl status errors	chown/chmod narrowly; reload service
SSH change locks you out	sshd -t; open a second session	Apply change; systemctl reload sshd; test on new session before closing existing
App reload fails	app -t (syntax test); journalctl -xeu app	Revert last edit; apply minimal fix; reload
Mounts missing	findmnt; dmesg -T; cat /etc/fstab	Correct fstab; test with mount -a (caution); revert if failures
Service up but unreachable	ss -tulpn; firewall list; routing	Add targeted firewall allow rule; persist; verify inbound reachability

## Representative mistakes and precise fixes Each subsection separates read-only diagnosis from safe state changes. ### 1) Systemd unit pitfalls - Read-only diagnosis - View effective unit, including drop-ins: systemctl cat my.service - Show runtime properties: systemctl show -p FragmentPath,ExecStart,Environment,User,Group my.service - Check logs: journalctl -xeu my.service - Verify syntax: systemd-analyze verify /etc/systemd/system/my.service - Common mistakes and fixes - Shell syntax in ExecStart: systemd does not invoke a shell by default. - Bad: ExecStart=/usr/bin/mycmd $VAR | tee /tmp/log - Good: set Environment=VAR=value and use a simple ExecStart, or explicitly ExecStart=/bin/bash -lc '...' - Edits ignored after changing unit files - Fix: systemctl daemon-reload; systemctl restart my.service - Overwriting vendor units - Safer: mkdir -p /etc/systemd/system/my.service.d; edit override.conf with only changed keys. - Rollback - Move the override out of the directory or restore the .bak; systemctl daemon-reload; systemctl restart my.service ### 2) Network routes and DNS - Read-only diagnosis - Addresses: ip -br addr - Routing decision: ip route get 8.8.8.8 - DNS resolution: getent hosts api.example.com; getent ahosts api.example.com - Socket check: ss -lntp | grep :443 - Common mistakes and fixes - Wrong default route or missing route for a subnet - Safe action: ip route replace default via GATEWAY dev IFACE (temporary); persist using your distro's network config tool. - DNS points to unexpected IP - Safe action: update resolvers (e.g., /etc/resolv.conf managed by systemd-resolved or netplan) and validate with getent; avoid editing generated files directly. - Rollback - Revert the network config file and netplan apply or equivalent; if remote, ensure out-of-band access before changing routes. ### 3) File ownership and permissions - Read-only diagnosis - Trace path permissions: namei -l /path/to/file - Inspect metadata: ls -l; stat file - Service logs will show permission denials - Common mistakes and fixes - Sensitive config world-readable or wrong owner - sshd_config example: chown root:root /etc/ssh/sshd_config; chmod 600 /etc/ssh/sshd_config; sshd -t - App fails to read its own state dir - Safe action: chown -R appuser:appgroup /var/lib/myapp with the minimal required scope. - Rollback - Restore saved modes with chmod and chown using recorded values; never blanket chmod -R 777. ### 4) SSH and sudo access - Read-only diagnosis - Validate sshd: sshd -t (syntax); sshd -T | sort (effective) - Check sudoers: visudo -c - Common mistakes and fixes - Lockout risk when changing SSH - Safe action: keep an existing root/session open; apply the change; systemctl reload sshd; test with a new connection before closing the old one. - Broken sudoers include or syntax - Safe action: only edit via visudo; correct the bad line; re-run visudo -c until clean. - Rollback - Restore sshd_config.bak and reload; for sudo, revert the last include in /etc/sudoers.d and validate. ### 5) Package or application configuration - Read-only diagnosis - Find the effective config: read app docs; often /etc/ or drop-ins in conf.d - Test syntax: nginx -t; haproxy -c -f /etc/haproxy/haproxy.cfg; php-fpm -t; named-checkconf - Dump full render when supported: nginx -T - Common mistakes and fixes - Editing the wrong file or wrong include order - Safe action: place changes in a numbered include (e.g., /etc/nginx/conf.d/20-feature.conf); verify include order with nginx -T; reload. - Multiple changes in one deploy - Safe action: ship one change, test, then proceed to the next. - Rollback - Remove or rename the include; restore last known-good file; reload. ### 6) Storage and mounts - Read-only diagnosis - Current mounts: findmnt -A - fstab: cat /etc/fstab - Disk and FS messages: dmesg -T | tail -n 100 - Common mistakes and fixes - Incorrect fstab options or device path - Safe action: fix the single line; test with mount -a (caution: run in a local console or with OOB access to avoid remote lockout). Consider nofail,x-systemd.automount for non-critical mounts. - Forgotten filesystem creation - Safe action: create with the correct tool (e.g., mkfs.xfs /dev/...), then add to fstab and mount - Rollback - umount the problematic mount; revert fstab backup; run mount -a to restore prior state. ### 7) Firewall rules - Read-only diagnosis - Listening sockets: ss -tulpn - Current rules: nft list ruleset (nftables) or iptables -S (legacy) - Common mistakes and fixes - Service listens but is blocked - Safe action: add a narrowly scoped allow rule for the port/protocol and source needed; persist with your firewall tool (firewalld, ufw, direct nftables). - Default drop without explicit return in a chain - Safe action: insert rule in the correct hook/chain before the drop; verify with a new connection. - Rollback - Remove the last rule or restore the previous ruleset snapshot; keep a copy of working rules before changes. ### 8) Environment-variable precedence - Read-only diagnosis - For systemd services: systemctl show -p Environment my.service; systemctl cat my.service (check EnvironmentFile=) - At shell: env | sort; printenv NAME - Common mistakes and fixes - Expecting interactive shell vars to affect systemd services - Fix: set Environment= or EnvironmentFile=/etc/default/myapp in the unit or its drop-in; systemctl daemon-reload; systemctl restart my.service - Conflicting settings across files - Action: dump effective config (systemctl cat); consolidate to a single source of truth. - Rollback - Remove or comment the added Environment settings; daemon-reload; restart. ## Pre-change, validation, rollback checklist

Phase	Do this	Evidence to keep
Pre-change	Backup the exact files; capture systemctl cat/show, ss -tulpn, ip -br addr, ip route, findmnt, and recent journal for the unit	Timestamped .bak files; command outputs saved to a ticket or log file
Validation	Run the correct syntax test (nginx -t, sshd -t, visudo -c, systemd-analyze verify); dry-run if supported	Exit code 0; screenshots or output snippets showing OK
Rollback	Revert the .bak; systemctl daemon-reload if units changed; reload/restart the affected service; verify sockets and logs	Post-rollback checks passing; diff shows restored config

## Practical examples: what to record and how to interpret it - systemctl status my.service: look for ExecStart, exit code, and recent errors. Non-zero exit statuses or timeout messages indicate bad arguments or permission issues. - journalctl -xeu my.service: parse the first failing line (often contains the file path or directive name that broke). - ip route get 1.1.1.1: confirms which interface and gateway the kernel would use; mismatches imply missing or wrong routes. - getent hosts name: if resolution fails or returns unexpected IPs, check resolvers and search domains. - ss -tulpn | grep :PORT: if empty, the service is not bound; re-check bind address and listen directives. - findmnt /mnt/target: if no entry, the mount is not active; check fstab and underlying device availability. ## Escalation criteria - You cannot reproduce the failure locally but remote users still fail (likely external network or LB). - Kernel or hardware errors appear in dmesg (disk I/O, NIC resets). - A rollback does not restore service to the prior baseline. - Multiple subsystems fail simultaneously after a change (broader change management or dependency issue). ## Operational runbook (reuse this) - Stabilize - Freeze changes; announce the incident. Start a console or out-of-band session. - Capture - Save unit files and app conf .bak; record systemctl cat/show/status, journalctl -xeu, ip -br addr, ip route, ss -tulpn, findmnt, getent outputs. - Diagnose - Identify the single owning subsystem; validate syntax with the relevant -t or verify command. - Change safely - Make one minimal, reversible change (prefer drop-ins or single include). systemctl daemon-reload if units changed; prefer reload over restart when supported. - Verify - Re-test the symptom; check logs and sockets; confirm remote reachability if applicable. - Roll back if red - Immediately restore the .bak; reload/restart; confirm recovery. Escalate if baseline does not return. - Document and prevent - Record root cause, the exact fix, and a test to prevent regression (pre-deploy syntax test, include order check, or permissions assertion). ## Conclusion Treat Linux configuration like code under change control: capture the current state, validate syntax, change one thing at a time, verify, and keep an instant rollback. With the commands and patterns above—systemctl cat/show/status, journalctl, ip -br addr, ip route get, getent, ss, sshd -t, visudo -c, findmnt, mount -a used cautiously—you can diagnose faster, avoid collateral damage, and recover confidently during real incidents.