Intro
Linux errors are inevitable, but how you respond determines whether an incident is a brief blip or a prolonged outage. This guide arms developers, DevOps consultants, and technical startup teams with a systematic approach to diagnosing and fixing common Linux issues. Rather than scattered tips, we focus on a repeatable workflow: observe before changing, limit the blast radius, use placeholders instead of secrets, verify every fix, and document recovery paths.
You will learn to capture the current state, identify the exact component behavior, apply the smallest justified change, and confirm the result with concrete commands and expected outputs. Every example includes version-specific considerations, prerequisites, and a tested rollback plan.
Version and Environment Inventory
Before touching anything, know exactly what you are working with. A mismatch between your assumptions and the actual environment is the root cause of many failed fixes.
Check the OS and kernel version:
cat /etc/os-release
uname -r
Expected output on Ubuntu 22.04:
VERSION_ID="22.04"
5.15.0-91-generic
For Red Hat-based systems:
cat /etc/redhat-release
Example output: Red Hat Enterprise Linux release 9.2 (Plow)
List installed packages relevant to your issue. For example, if you are debugging a Docker networking error, confirm Docker's version and storage driver:
docker version
sudo docker info | grep -E "Storage Driver|Server Version"
Sample output:
Server Version: 24.0.7
Storage Driver: overlay2
Capture systemd units that are failing or degraded:
systemctl --failed --no-pager
Output shows failed units, such as:
UNIT LOAD ACTIVE SUB DESCRIPTION
● docker.service loaded failed failed Docker Application Container Engine
Always record a timestamp before you make changes, so you can correlate log entries:
date -u +"%Y-%m-%dT%H:%M:%SZ"
Blast radius note: All commands in this section are read-only. They do not alter system state and are safe to run on production systems.
Safe Configuration Path
Configuration errors are among the most common Linux pitfalls. The key is to never edit a file blindly. Always back up the original, make a minimal change, test syntax if available, and know how to revert.
1. Back up the configuration file with a timestamp:
sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak.$(date +%Y%m%d%H%M%S)
2. Validate the syntax before applying. For nginx:
sudo nginx -t
Expected success:
nginx: configuration file /etc/nginx/nginx.conf test is successful
For SSH daemon:
sudo sshd -t
No output means syntax is valid.
3. Reload or restart the service with minimal disruption. Prefer reload when supported:
sudo systemctl reload nginx
If reload fails, check status immediately:
sudo systemctl status nginx --no-pager
4. If the service fails, roll back to the backup:
sudo cp /etc/nginx/nginx.conf.bak.20250219T143000 /etc/nginx/nginx.conf
sudo systemctl restart nginx
Example: Fixing a common SSH daemon misconfiguration. Suppose SSH is refusing password authentication because PasswordAuthentication is set to no inadvertently. You edit /etc/ssh/sshd_config:
PasswordAuthentication yes
Then validate and restart:
sudo sshd -t && sudo systemctl restart sshd
Verify the setting took effect:
sudo sshd -T | grep -i passwordauthentication
Expected output:
passwordauthentication yes
Security note: Never put real passwords, tokens, or private keys in configuration examples. Use clearly labeled placeholders like YOUR_SECRET_HERE in docs, but in real files manage secrets with a vault or environment variables.
Verification and Diagnostics
A fix is not complete until you verify the original symptom is gone and the system behaves as expected. This section covers essential diagnostic commands for common Linux error categories.
Diagnosing File System and Disk Errors
Check disk space and inode usage:
df -h /var
sudo du -sh /var/log/* | sort -rh | head -5
Sample output:
2.1G /var/log
1.1G /var/log/syslog
...
If inodes are exhausted, df -h may show space available but file creation fails. Check inodes:
df -i /var
Check for file system corruption or errors on the next boot. Use fsck with care, only on unmounted partitions:
sudo umount /dev/sdb1
sudo fsck -y /dev/sdb1
Expected output ends with FILE SYSTEM CLEAN or lists corrections made.
Diagnosing Process and Memory Issues
Find processes consuming excessive CPU or memory:
ps aux --sort=-%mem | head -10
Or use top interactively. In batch mode:
top -bn1 | head -20
Check for zombie processes:
ps aux | awk '$8=="Z" {print}'
If zombies accumulate, find their parent and kill it:
ps -o ppid= -p <ZOMBIE_PID>
Diagnosing Network Errors
Test connectivity and DNS resolution:
ping -c 4 8.8.8.8
nslookup example.com
If ping fails but DNS resolves, check routing:
ip route show default
Check listening ports and processes using them:
sudo ss -tulpn | grep LISTEN
Sample output:
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1123,fd=3))
Example: Debugging a Docker Container Exit Code 137
Exit code 137 often means the container was killed due to OOM (out-of-memory) or a stop signal. Verify with:
docker inspect --format='{{.State.ExitCode}} {{.State.OOMKilled}}' mycontainer
Expected: 137 true if OOM, 137 false if stopped externally. Check system logs for OOM killer messages:
sudo journalctl -k | grep -i oom
Sample output:
kernel: Out of memory: Killed process 2312 (node) total-vm:1234567kB, anon-rss:900000kB
Always compare observed output against expected output. Document the expected signal before running diagnostics.
Failure Modes and Recovery
Anticipate what can go wrong and have a recovery plan ready before you make a change. This section covers common failure modes and step-by-step recovery examples.
Failure: Systemd Service Fails to Start
Symptom: systemctl start myservice returns error code, and systemctl status shows failed.
Diagnose:
sudo journalctl -u myservice.service -n 50 --no-pager
Look for the root cause. For example, if it says Executable path is not absolute, the unit file uses a relative path.
Fix: Edit the unit file (e.g., /etc/systemd/system/myservice.service) and set an absolute ExecStart:
ExecStart=/usr/local/bin/myservice --config /etc/myservice/config.yaml
Reload systemd and restart:
sudo systemctl daemon-reload
sudo systemctl start myservice
Verify:
sudo systemctl is-active myservice
Expected: active.
Rollback: If the service still fails, revert the unit file from backup or reinstall the package.
Failure: Disk Full Due to Rotated Logs
Symptom: Application writes fail with No space left on device, but df -h shows space available.
Diagnose: Check deleted but still open file handles:
sudo lsof +L1 | grep deleted
Output shows processes holding deleted files. For example:
nginx 1234 www-data 12u REG 253,0 1048576 917504 /var/log/nginx/access.log (deleted)
Fix: Gracefully reload or restart the process holding the handle:
sudo systemctl reload nginx
The space will be reclaimed. Confirm:
df -h /
Prevention: Use log rotation with copytruncate or send logs to a central syslog.
Failure: DNS Resolution Intermittent
Symptom: Some commands fail with Temporary failure in name resolution, while others succeed.
Diagnose: Check /etc/resolv.conf and systemd-resolved status:
cat /etc/resolv.conf
sudo systemctl status systemd-resolved --no-pager
If using NetworkManager, check connection DNS settings:
nmcli device show eth0 | grep IP4.DNS
Fix: Correct DNS server order or add a fallback. For example, edit /etc/netplan/01-netcfg.yaml (Ubuntu 20.04+):
network:
version: 2
ethernets:
eth0:
dhcp4: true
nameservers:
addresses: [1.1.1.1, 8.8.8.8]
Apply with:
sudo netplan apply
Verify:
resolvectl status
Or test with a dig command.
Rollback: Restore the original netplan file or reconfigure via DHCP if applicable.
Operations Checklist
Use this checklist before and after any change in a production-like environment. Adapt it to your specific component, but ensure every item is addressed.
Pre-Change Checklist
- [ ] Identify the exact component and its version (e.g.,
nginx -v,php -v). - [ ] Capture current state with read-only commands and save output to a text file.
- [ ] Record the current time (UTC) to correlate logs.
- [ ] Identify the expected result and failure signal.
- [ ] Prepare the smallest possible change with a clear rollback plan.
- [ ] Confirm you have sufficient privileges (sudo -l) and that you are on the correct host (hostname).
- [ ] If changing configuration, back up the original file with a timestamp suffix.
Post-Change Verification Checklist
- [ ] Run the service's syntax check if available (e.g.,
nginx -t,sshd -t). - [ ] Reload or restart the service using the least disruptive method.
- [ ] Check service status:
systemctl is-active <service>. - [ ] Verify the original symptom is gone: run the original failing command and compare output.
- [ ] Tail recent logs for errors:
journalctl -u <service> -n 50 --no-pager. - [ ] Monitor for a few minutes to ensure stability (e.g.,
watch -n 2 systemctl status <service>). - [ ] Document the change and its outcome in your incident log or change management system.
Concrete Example: Applying the Checklist to a Web Server TLS Certificate Update
Suppose you need to update an expired TLS certificate for nginx.
Pre-change:
- Confirm nginx version:
nginx -v→nginx/1.22.1. - Check current certificate expiry:
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates. - Back up old cert and key:
sudo cp /etc/ssl/certs/example.com.crt /etc/ssl/certs/example.com.crt.oldand same for key. - Place new cert and key files, then run
sudo nginx -t→ expectsyntax is ok.
Post-change:
- Reload nginx:
sudo systemctl reload nginx. - Check status:
systemctl is-active nginx→active. - Verify new expiry:
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -enddate. - Check logs:
tail -f /var/log/nginx/error.logfor any TLS errors. - Update documentation: record new expiry date and file locations.
Conclusion
Mastering Linux common errors is less about memorizing fixes and more about adopting a disciplined workflow. Start with a precise version and environment inventory, capture observable state before touching anything, make minimal, reversible changes, and always verify the outcome against a predefined expected signal. Use placeholders in documentation and keep real secrets out of command examples and log files.
Choose one low-risk verification from this guide and apply it today. For instance, run systemctl --failed to see if any services are currently in a failed state, or check your most critical service's logs with journalctl -u <service> -n 50. Record what you find, compare it to expectations, and decide on a next step. Over time, this practice builds a robust operational muscle memory, turning errors from crises into manageable checkpoints.
Remember to review dependencies such as systemd, Bash, and Docker only when they directly affect your issue. Keep your recovery paths documented before an incident forces a decision. With diligence and the right tools, you can keep your Linux systems healthy and resilient.