Linux Troubleshooting with Practical Examples: A Field Guide for DevOps

Introduction

When a production Linux server misbehaves, the difference between a 10-minute fix and a 10-hour outage often comes down to having a structured troubleshooting methodology. This guide is built for developers, DevOps engineers, and technical startup teams who need to move from symptom to root cause quickly and safely. We will cover the core areas: gathering environment information, making safe configuration changes, running diagnostics, handling typical failure modes, and maintaining an operational checklist.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. Every step follows a disciplined pattern:

1. Identify the component and its version.
2. Observe the current state with read-only commands.
3. Define the expected result and failure signal.
4. Change one scoped item only when necessary.
5. Verify the outcome with a concrete command.
6. Document the recovery path before you need it.

We will use explicit placeholders in commands (like your_service_name) and never include real credentials.

Version and Environment Inventory

Before you can fix anything, you must know exactly what you are running. This section helps you collect the essential facts about your system.

Identify the Linux Distribution and Kernel

Start with the basics: which distribution and kernel are you on? Different distributions package software differently and have different default configurations.

# Show distribution information
cat /etc/os-release

Expected output (excerpt):

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"

Check the kernel version:

uname -r

Expected output:

5.4.0-155-generic

Record these values because many troubleshooting guides and tools have version-specific behaviors. For example, certain kernel parameters or system call behavior may differ between kernel versions.

Check System Resources

A slow system is often a resource-constrained system. Check memory, CPU, and disk usage.

# Memory usage in megabytes
free -m

Expected output (excerpt):

              total        used        free      shared  buff/cache   available
Mem:           7984        1234        4500         120        2250        6200
Swap:          2048           0        2048

Look at the available column; if it is near zero, the system is under memory pressure.

# CPU load averages
uptime

Expected output:

 14:23:45 up 21 days,  4:12,  2 users,  load average: 0.85, 0.92, 1.10

Load averages above the number of CPU cores (check with nproc) indicate sustained overload.

# Disk usage
 df -h

Expected output (excerpt):

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        98G   55G   38G  60% /

A filesystem at 90% or more needs attention; some services fail when the disk is full.

List Key Services and Ports

Identify what is supposed to be running and whether it is listening on expected ports.

# List running services (on systemd)
systemctl list-units --type=service --state=running

Find a specific service status:

systemctl status nginx

If it is not running, start it and enable it to run at boot:

sudo systemctl start nginx
sudo systemctl enable nginx

Check listening ports:

sudo ss -tulpn | grep nginx

Expected output:

tcp   LISTEN 0      128          0.0.0.0:80        0.0.0.0:*    users:(("nginx",pid=1234,fd=6))
tcp   LISTEN 0      128             [::]:80           [::]:*    users:(("nginx",pid=1234,fd=7))

If a service should be listening but is not, check its logs and configuration.

Safe Configuration Path

Changing configuration files can break things if done carelessly. Follow a careful process: back up, edit with version control, validate syntax, and reload gracefully.

Backup Configuration Files

Always make a timestamped backup before modifying any configuration file.

sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak.$(date +%Y%m%d)

Verify the backup exists:

ls -l /etc/nginx/nginx.conf.bak.*

Edit with Version Control

For critical configurations, keep them under version control. If not already, initialize a Git repository in /etc (with appropriate permissions) and commit changes.

cd /etc
sudo git init
sudo git add nginx/nginx.conf
sudo git commit -m "Initial commit of nginx config"

After editing, commit the change:

sudo git add nginx/nginx.conf
sudo git commit -m "Updated worker_processes to 4"

This allows you to roll back easily with git checkout.

Validate Syntax Before Applying

Most daemons provide a syntax check command.

For Nginx:

sudo nginx -t

Expected output on success:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

For Apache:

sudo apachectl configtest

For SSH daemon:

sudo sshd -t

If the syntax check fails, fix the errors before reloading or restarting.

Reload Instead of Restart When Possible

A reload applies configuration changes without dropping existing connections. For services that support it (like Nginx, Apache, HAProxy), use reload:

sudo systemctl reload nginx

Check the service status after reload:

systemctl status nginx

If the service fails to reload, check logs and consider rolling back to the backup:

sudo cp /etc/nginx/nginx.conf.bak.20250101 /etc/nginx/nginx.conf
sudo systemctl reload nginx

Verification and Diagnostics

After making a change or when troubleshooting an issue, you need to verify that the system is behaving as expected and collect diagnostic information.

Check Logs

Logs are the first place to look for error messages.

# View the last 50 lines of the system log
sudo journalctl -n 50

For a specific service:

sudo journalctl -u nginx -n 50 --no-pager

Follow logs in real time:

sudo journalctl -u nginx -f

Older syslog files may be in /var/log/:

sudo tail -n 50 /var/log/nginx/error.log

Test Network Connectivity

Use curl to test an HTTP endpoint:

curl -I http://localhost

Expected output on success:

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)

If you see a connection refused or timeout, the service may not be listening or a firewall is blocking.

Check DNS resolution:

dig example.com

If dig is not installed, install dnsutils or use nslookup (deprecated) or getent hosts example.com.

Test Local Service with Telnet or Netcat

For non-HTTP services, use telnet or nc to check if a port is open and responding.

# Test SSH port 22
nc -zv localhost 22

Expected output:

Connection to localhost 22 port [tcp/ssh] succeeded!

Use Strace for System Calls

If a process is hanging or failing mysteriously, strace can show what system calls it is making.

sudo strace -p 1234

Replace 1234 with the PID of the process. Look for errors like ENOENT (file not found) or EACCES (permission denied).

Failure Modes and Recovery

Common failure modes have standard recovery procedures. Here we cover disk full, service crash, and misconfiguration.

Disk Full

A full disk can cause various failures: services unable to write logs, databases refusing writes, and system instability.

Symptoms:

• df -h shows Use% at 100%.
• Error messages like "No space left on device" in logs.
• Services crashing or becoming unresponsive.

Recovery steps:

1. Identify large files or directories:

sudo du -ah / | sort -rh | head -n 20

1. Check for deleted but still open files (common with log rotation):

sudo lsof +L1

1. Clean up package cache:

sudo apt clean

1. Remove old log files (but consider logrotate configuration):

sudo find /var/log -type f -name "*.log" -mtime +30 -delete

1. If possible, expand the filesystem or add disk space.

Verification:

df -h

Ensure Use% is below 90% and critical services are running.

Service Crash Loop

A service that keeps crashing may be misconfigured or hitting a resource limit.

Symptoms:

• systemctl status your_service shows Active: activating (auto-restart) or failed.
• Repeated error messages in logs.

Recovery steps:

1. Check the service logs for the exact error:

sudo journalctl -u your_service -n 100 --no-pager

1. Check if the service is trying to bind to a port already in use:

sudo ss -tulpn | grep your_port

1. Check resource limits:

ulimit -a

1. If it is an OOM (Out Of Memory) kill, check dmesg:

dmesg | grep -i oom

1. Fix the root cause (e.g., increase memory limit, correct config), then start the service:

sudo systemctl start your_service

Verification:

systemctl status your_service

Should show Active: active (running).

Misconfiguration Rollback

If a recent configuration change caused the failure, roll back to the previous version.

If using Git:

cd /etc
sudo git log --oneline -5
sudo git checkout <commit_hash> -- nginx/nginx.conf
sudo systemctl reload nginx

If using backup files:

sudo cp /etc/nginx/nginx.conf.bak.20250101 /etc/nginx/nginx.conf
sudo systemctl reload nginx

Verification:

Check that the service is running and the original behavior is restored.

Operations Checklist

Maintain a checklist for routine checks and troubleshooting. Here is a table of essential items, the owner, and frequency.

Each item should have a clear owner (not a group) and a defined frequency. For example, the DevOps Engineer checks disk usage daily; the on-call engineer reviews errors each shift.

Common Pitfalls and How to Avoid Them

Even experienced engineers make mistakes. Here are common pitfalls in Linux troubleshooting and how to recover.

1. Making Changes Without a Backup

Why it happens: Urgency, overconfidence, or lack of discipline.

How to avoid: Always create a timestamped backup before editing. Use configuration management tools (Ansible, Puppet, Chef) to maintain known-good states.

Recovery: If you forgot, check if the system has automatic backups (e.g., /etc snapshots). Otherwise, reconstruct the file from documentation or a fresh install template.

2. Running Commands with Incorrect Permissions

Why it happens: Trying to read or write files as the wrong user.

How to avoid: Use sudo only when necessary, and verify file permissions with ls -l.

Recovery: If you changed permissions incorrectly, restore them from backup or use chmod and chown to set the correct values (refer to documentation).

3. Misinterpreting Log Messages

Why it happens: Logs can be verbose, and error messages may be misleading.

How to avoid: Learn to filter logs with grep, journalctl options, and correlate with timestamps. Understand the context: a warning may be harmless, while an error may be fatal.

Recovery: If you acted on a misinterpreted log, verify the actual system state with other commands and roll back if necessary.

4. Ignoring Version Differences

Why it happens: Copying commands from online sources without checking if they apply to your distribution or version.

How to avoid: Always check the OS version and software version before applying commands. Use the official documentation for your specific version.

Recovery: If a command caused issues due to version mismatch, revert the change and consult the correct documentation.

Conclusion

Linux troubleshooting is a systematic discipline. By following the patterns outlined in this guide, you can reduce downtime and avoid costly mistakes. Remember to:

• Inventory your environment before making changes.
• Use read-only commands to observe.
• Back up and validate configuration changes.
• Verify outcomes with concrete commands.
• Document recovery procedures.

As a next step, apply this methodology to a low-risk component in your environment. Run through the Version and Environment Inventory, identify a safe configuration change, and practice the rollback procedure. With repetition, these habits become second nature, and you will be well-prepared for the inevitable production incident.