## Introduction When a production Linux server misbehaves, the difference between a 10-minute fix and a 10-hour outage often comes down to having a structured troubleshooting methodology. This guide is built for developers, DevOps engineers, and technical startup teams who need to move from symptom to root cause quickly and safely. We will cover the core areas: gathering environment information, making safe configuration changes, running diagnostics, handling typical failure modes, and maintaining an operational checklist. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. Every step follows a disciplined pattern: - Identify the component and its version. - Observe the current state with read-only commands. - Define the expected result and failure signal. - Change one scoped item only when necessary. - Verify the outcome with a concrete command. - Document the recovery path before you need it. We will use explicit placeholders in commands (like your_service_name ) and never include real credentials. ## Version and Environment Inventory Before you can fix anything, you must know exactly what you are running. This section helps you collect the essential facts about your system. ### Identify the Linux Distribution and Kernel Start with the basics: which distribution and kernel are you on? Different distributions package software differently and have different default configurations. # Show distribution information cat /etc/os-release Expected output (excerpt): NAME="Ubuntu" VERSION="20.04.6 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.6 LTS" VERSION_ID="20.04" Check the kernel version: uname -r Expected output: 5.4.0-155-generic Record these values because many troubleshooting guides and tools have version-specific behaviors. For example, certain kernel parameters or system call behavior may differ between kernel versions. ### Check System Resources A slow system is often a resource-constrained system. Check memory, CPU, and disk usage. # Memory usage in megabytes free -m Expected output (excerpt): total used free shared buff/cache available Mem: 7984 1234 4500 120 2250 6200 Swap: 2048 0 2048 Look at the available column; if it is near zero, the system is under memory pressure. # CPU load averages uptime Expected output: 14:23:45 up 21 days, 4:12, 2 users, load average: 0.85, 0.92, 1.10 Load averages above the number of CPU cores (check with nproc ) indicate sustained overload. # Disk usage df -h Expected output (excerpt): Filesystem Size Used Avail Use% Mounted on /dev/sda1 98G 55G 38G 60% / A filesystem at 90% or more needs attention; some services fail when the disk is full. ### List Key Services and Ports Identify what is supposed to be running and whether it is listening on expected ports. # List running services (on systemd) systemctl list-units --type=service --state=running Find a specific service status: systemctl status nginx If it is not running, start it and enable it to run at boot: sudo systemctl start nginx sudo systemctl enable nginx Check listening ports: sudo ss -tulpn | grep nginx Expected output: tcp LISTEN 0 128 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1234,fd=6)) tcp LISTEN 0 128 [::]:80 [::]:* users:(("nginx",pid=1234,fd=7)) If a service should be listening but is not, check its logs and configuration. ## Safe Configuration Path Changing configuration files can break things if done carelessly. Follow a careful process: back up, edit with version control, validate syntax, and reload gracefully. ### Backup Configuration Files Always make a timestamped backup before modifying any configuration file. sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak.$(date +%Y%m%d) Verify the backup exists: ls -l /etc/nginx/nginx.conf.bak.* ### Edit with Version Control For critical configurations, keep them under version control. If not already, initialize a Git repository in /etc (with appropriate permissions) and commit changes. cd /etc sudo git init sudo git add nginx/nginx.conf sudo git commit -m "Initial commit of nginx config" After editing, commit the change: sudo git add nginx/nginx.conf sudo git commit -m "Updated worker_processes to 4" This allows you to roll back easily with git checkout . ### Validate Syntax Before Applying Most daemons provide a syntax check command. For Nginx: sudo nginx -t Expected output on success: nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful For Apache: sudo apachectl configtest For SSH daemon: sudo sshd -t If the syntax check fails, fix the errors before reloading or restarting. ### Reload Instead of Restart When Possible A reload applies configuration changes without dropping existing connections. For services that support it (like Nginx, Apache, HAProxy), use reload : sudo systemctl reload nginx Check the service status after reload: systemctl status nginx If the service fails to reload, check logs and consider rolling back to the backup: sudo cp /etc/nginx/nginx.conf.bak.20250101 /etc/nginx/nginx.conf sudo systemctl reload nginx ## Verification and Diagnostics After making a change or when troubleshooting an issue, you need to verify that the system is behaving as expected and collect diagnostic information. ### Check Logs Logs are the first place to look for error messages. # View the last 50 lines of the system log sudo journalctl -n 50 For a specific service: sudo journalctl -u nginx -n 50 --no-pager Follow logs in real time: sudo journalctl -u nginx -f Older syslog files may be in /var/log/ : sudo tail -n 50 /var/log/nginx/error.log ### Test Network Connectivity Use curl to test an HTTP endpoint: curl -I http://localhost Expected output on success: HTTP/1.1 200 OK Server: nginx/1.18.0 (Ubuntu) If you see a connection refused or timeout, the service may not be listening or a firewall is blocking. Check DNS resolution: dig example.com If dig is not installed, install dnsutils or use nslookup (deprecated) or getent hosts example.com . ### Test Local Service with Telnet or Netcat For non-HTTP services, use telnet or nc to check if a port is open and responding. # Test SSH port 22 nc -zv localhost 22 Expected output: Connection to localhost 22 port [tcp/ssh] succeeded! ### Use Strace for System Calls If a process is hanging or failing mysteriously, strace can show what system calls it is making. sudo strace -p 1234 Replace 1234 with the PID of the process. Look for errors like ENOENT (file not found) or EACCES (permission denied). ## Failure Modes and Recovery Common failure modes have standard recovery procedures. Here we cover disk full, service crash, and misconfiguration. ### Disk Full A full disk can cause various failures: services unable to write logs, databases refusing writes, and system instability. Symptoms: - df -h shows Use% at 100%. - Error messages like "No space left on device" in logs. - Services crashing or becoming unresponsive. Recovery steps: - Identify large files or directories: sudo du -ah / | sort -rh | head -n 20 - Check for deleted but still open files (common with log rotation): sudo lsof +L1 - Clean up package cache: sudo apt clean - Remove old log files (but consider logrotate configuration): sudo find /var/log -type f -name "*.log" -mtime +30 -delete - If possible, expand the filesystem or add disk space. Verification: df -h Ensure Use% is below 90% and critical services are running. ### Service Crash Loop A service that keeps crashing may be misconfigured or hitting a resource limit. Symptoms: - systemctl status your_service shows Active: activating (auto-restart) or failed . - Repeated error messages in logs. Recovery steps: - Check the service logs for the exact error: sudo journalctl -u your_service -n 100 --no-pager - Check if the service is trying to bind to a port already in use: sudo ss -tulpn | grep your_port - Check resource limits: ulimit -a - If it is an OOM (Out Of Memory) kill, check dmesg : dmesg | grep -i oom - Fix the root cause (e.g., increase memory limit, correct config), then start the service: sudo systemctl start your_service Verification: systemctl status your_service Should show Active: active (running) . ### Misconfiguration Rollback If a recent configuration change caused the failure, roll back to the previous version. If using Git: cd /etc sudo git log --oneline -5 sudo git checkout -- nginx/nginx.conf sudo systemctl reload nginx If using backup files: sudo cp /etc/nginx/nginx.conf.bak.20250101 /etc/nginx/nginx.conf sudo systemctl reload nginx Verification: Check that the service is running and the original behavior is restored. ## Operations Checklist Maintain a checklist for routine checks and troubleshooting. Here is a table of essential items, the owner, and frequency.
CheckCommandExpected ResultOwnerFrequency
Disk usagedf -hAll filesystems < 80%DevOps EngineerDaily
Memory usagefree -mAvailable memory > 20%DevOps EngineerDaily
CPU loaduptimeLoad < number of coresDevOps EngineerDaily
Service statussystemctl status critical-serviceActive (running)System AdministratorHourly (or via monitoring)
Log errorsjournalctl -p err --since todayNo unexpected errorsOn-call EngineerEach shift
Backup statusCheck backup tool logsSuccessful backupsBackup AdministratorDaily
Each item should have a clear owner (not a group) and a defined frequency. For example, the DevOps Engineer checks disk usage daily; the on-call engineer reviews errors each shift. ## Common Pitfalls and How to Avoid Them Even experienced engineers make mistakes. Here are common pitfalls in Linux troubleshooting and how to recover. ### 1. Making Changes Without a Backup Why it happens: Urgency, overconfidence, or lack of discipline. How to avoid: Always create a timestamped backup before editing. Use configuration management tools (Ansible, Puppet, Chef) to maintain known-good states. Recovery: If you forgot, check if the system has automatic backups (e.g., /etc snapshots). Otherwise, reconstruct the file from documentation or a fresh install template. ### 2. Running Commands with Incorrect Permissions Why it happens: Trying to read or write files as the wrong user. How to avoid: Use sudo only when necessary, and verify file permissions with ls -l . Recovery: If you changed permissions incorrectly, restore them from backup or use chmod and chown to set the correct values (refer to documentation). ### 3. Misinterpreting Log Messages Why it happens: Logs can be verbose, and error messages may be misleading. How to avoid: Learn to filter logs with grep , journalctl options, and correlate with timestamps. Understand the context: a warning may be harmless, while an error may be fatal. Recovery: If you acted on a misinterpreted log, verify the actual system state with other commands and roll back if necessary. ### 4. Ignoring Version Differences Why it happens: Copying commands from online sources without checking if they apply to your distribution or version. How to avoid: Always check the OS version and software version before applying commands. Use the official documentation for your specific version. Recovery: If a command caused issues due to version mismatch, revert the change and consult the correct documentation. ## Conclusion Linux troubleshooting is a systematic discipline. By following the patterns outlined in this guide, you can reduce downtime and avoid costly mistakes. Remember to: - Inventory your environment before making changes. - Use read-only commands to observe. - Back up and validate configuration changes. - Verify outcomes with concrete commands. - Document recovery procedures. As a next step, apply this methodology to a low-risk component in your environment. Run through the Version and Environment Inventory, identify a safe configuration change, and practice the rollback procedure. With repetition, these habits become second nature, and you will be well-prepared for the inevitable production incident.