## Intro Linux performance tuning is the practice of systematically identifying and resolving bottlenecks to improve system responsiveness and throughput. This guide provides practical, hands-on techniques for developers, DevOps engineers, and technical teams who need to diagnose and optimize Linux systems. It covers essential commands, real-world examples, and a safe methodology to observe, change, and verify improvements. The focus is on operational safety: always observe the current state before making changes, understand the blast radius of any modification, use placeholders instead of real secrets, verify results, and document recovery steps. This approach minimizes risk and ensures that performance improvements are measurable and reversible. ## Version and Environment Inventory Before any tuning, document the environment. This includes the Linux distribution, kernel version, hardware details, and relevant software versions. Knowing the exact versions is critical because tuning parameters and tool behavior can vary significantly across releases. Prerequisites - Access to a Linux system with sudo or root privileges for system-level changes. - Basic familiarity with the command line and text editors. - A clear understanding of the application workload (e.g., web server, database). Commands to Capture Current State # Distribution and version cat /etc/os-release # Kernel version uname -r # CPU information lscpu # Memory information free -h # Disk and filesystem layout lsblk # Network interfaces ip -brief address Example Output (excerpt): $ uname -r 5.15.0-91-generic $ free -h total used free shared buff/cache available Mem: 15Gi 2.1Gi 11Gi 234Mi 1.8Gi 12Gi Swap: 2.0Gi 0B 2.0Gi This inventory establishes a baseline. Always timestamp your observations (e.g., date >> baseline.txt ) to correlate changes with performance metrics over time. Blast Radius and Recovery When changing system settings, know which files or services are affected. For example, editing /etc/sysctl.conf affects kernel parameters globally. Always back up the original file: sudo cp /etc/sysctl.conf /etc/sysctl.conf.bak-$(date +%Y%m%d) If a change causes issues, restore the backup and reboot or reload the configuration. ## Safe Configuration Path Tuning should follow a safe, incremental process: - Identify the bottleneck using monitoring tools (see Verification and Diagnostics). - Research the appropriate parameter for your kernel version and workload. - Make one change at a time in a test environment first. - Apply the change using the appropriate method (e.g., sysctl, systemd unit, config file). - Verify the effect with benchmarks or production metrics. - Document the change and its rollback procedure. Example: Adjusting Swappiness Swappiness controls how aggressively the kernel swaps memory pages to disk. A lower value keeps more data in RAM, which can improve performance for memory-intensive applications, but may increase OOM risk if RAM is exhausted. Observation Check current swappiness: cat /proc/sys/vm/swappiness Default is often 60. For a database server, you might want 10. Change (temporary, until reboot) sudo sysctl vm.swappiness=10 Permanent change Add to /etc/sysctl.conf or a file in /etc/sysctl.d/ : vm.swappiness=10 Then apply: sudo sysctl -p Verification cat /proc/sys/vm/swappiness # Output: 10 Recovery If the change causes instability (e.g., OOM killer activates), revert by setting the value back to 60 and removing the line from the config file. ## Verification and Diagnostics Diagnosing performance issues requires a methodical approach. Start with high-level tools to identify the subsystem under pressure, then drill down. System-Wide Monitoring - top / htop : Real-time view of CPU and memory usage. Look for processes with high %CPU or %MEM. - vmstat : Reports virtual memory statistics, CPU activity, and I/O. vmstat 1 5 # 1-second interval, 5 samples Output columns: r (run queue), b (blocked), si / so (swap in/out), us / sy / id (user/system/idle CPU). - iostat : Disk I/O statistics. iostat -x 1 5 Look for high %util (device saturation) and high await (I/O latency). - sar : Collects and reports system activity. Install sysstat if not present. sar -u 1 5 # CPU usage sar -r 1 5 # Memory usage sar -n DEV 1 5 # Network devices CPU Bottleneck Diagnosis If vmstat shows high r (run queue) and high us (user CPU), a process is CPU-bound. Identify the process with top and then use perf to profile it: sudo perf top # Or for a specific PID: sudo perf record -p -g -- sleep 10 sudo perf report Memory Bottleneck Diagnosis High si / so (swap activity) indicates memory pressure. Use free -h to check available memory and ps aux --sort=-%mem | head to find memory hogs. Also check for memory leaks with tools like valgrind or pmap . I/O Bottleneck Diagnosis If iostat shows high %util and await , the disk is a bottleneck. Use iotop to see which processes are generating I/O. Consider optimizing with faster storage, adjusting I/O scheduler, or using a RAM cache. Network Bottleneck Diagnosis Use ss -s for socket statistics and nload or iftop for bandwidth monitoring. High packet loss or retransmissions can be checked with netstat -s or ss -ti . Worked Example: Detecting CPU Throttling Suppose an application is slow. Run top and observe %Cpu(s): 99.0 us . Then run vmstat 1 5 : procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 4 0 0 1200000 200000 300000 0 0 0 10 500 1000 99 1 0 0 0 4 0 0 1190000 200000 300000 0 0 0 0 500 1100 99 1 0 0 0 The high r (4) and us (99) confirm CPU saturation. Identify the process and profile with perf to find hot functions. ## Failure Modes and Recovery Even with careful tuning, failures can occur. Understanding common failure modes helps you recover quickly. Configuration Error A typo in a config file can prevent a service from starting. Always validate configuration syntax before applying: # For systemd unit files: systemd-analyze verify /etc/systemd/system/myapp.service # For sysctl settings, check for errors when applying: sudo sysctl -p # For nginx: sudo nginx -t Kernel Panic or Boot Failure If a change to boot parameters (e.g., in GRUB) prevents booting, use a rescue disk or boot into a previous kernel from the GRUB menu. Always keep a known-good kernel installed. Resource Exhaustion Tuning for performance may inadvertently exhaust resources. For example, setting vm.overcommit_memory=2 strictly limits memory overcommit; if an application requests more memory than available, it may fail. Monitor with dmesg for OOM killer messages: dmesg | grep -i "out of memory" Recovery: adjust the parameter or increase RAM/swap. Network Misconfiguration Changing TCP parameters (e.g., net.ipv4.tcp_fin_timeout ) may cause connections to linger or drop prematurely. Always test network changes in a staging environment and monitor with ss -s and netstat -s . Rollback Strategy - Keep backups of all modified configuration files with timestamps. - Document the exact change and its expected effect. - Use configuration management tools (Ansible, Puppet) to version control changes. - For kernel parameters, note that temporary changes are cleared on reboot, so a reboot can be a quick rollback if permanent changes cause issues. ## Operations Checklist Use this checklist before and after making any performance tuning change. Pre-Change Checklist
ItemDescriptionOwnerFrequency
Backup current configurationCopy relevant files (e.g., /etc/sysctl.conf, /etc/nginx/nginx.conf) with timestamp.Sysadmin (Priya Shah)Every change
Record baseline metricsUse sar/vmstat/iostat to capture CPU, memory, I/O, network for at least 10 minutes.Performance Engineer (Carlos Gomez)Before each change
Identify rollback procedureDocument exact commands to revert (e.g., restore backup, reboot).Sysadmin (Priya Shah)Every change
Test in stagingApply change in a non-production environment and run load tests.QA Lead (Aisha Khan)Every change
Post-Change Verification
ItemDescriptionOwnerFrequency
Compare metricsRun same monitoring as baseline and compare with previous metrics.Performance Engineer (Carlos Gomez)After change
Check logs for errorsReview system logs (/var/log/syslog, application logs) for warnings.Sysadmin (Priya Shah)After change
Validate rollback worksIn staging, perform rollback and ensure system returns to previous state.QA Lead (Aisha Khan)After change
This checklist ensures that changes are safe, reversible, and monitored by accountable individuals. ## Common Pitfalls and How to Avoid Them Why it happens : Desire to fix multiple issues quickly. How to avoid : Change one parameter, measure, then proceed. Use a controlled experiment approach. - Changing Too Many Parameters at Once Why it happens : Assuming you know the system's normal state. How to avoid : Always collect baseline metrics before tuning. Without a baseline, you cannot prove improvement or diagnose regression. - Ignoring Baseline Data Why it happens : Guessing based on symptoms rather than metrics. How to avoid : Use systematic monitoring (vmstat, iostat, sar) to pinpoint the actual bottleneck. For example, high CPU may be caused by I/O wait, not CPU-bound processing. - Tuning for the Wrong Bottleneck Why it happens : Overconfidence or time pressure. How to avoid : Apply changes temporarily first (e.g., sysctl -w ), test, then make permanent if successful. - Permanent Changes Without Testing Why it happens : Underestimating future troubleshooting needs. How to avoid : Maintain a change log with date, parameter, old value, new value, reason, and rollback steps. - Not Documenting Changes Why it happens : Performance tuning often focuses solely on speed. How to avoid : Ensure that changes do not weaken security (e.g., disabling SYN cookies to improve connection rate may expose to SYN floods). - Neglecting Security Implications ## Conclusion Linux performance tuning is a disciplined process that requires observation, careful change, and verification. By following the safe configuration path, using the right diagnostic tools, and adhering to an operations checklist, you can improve system performance while minimizing risk. Always document changes, test in staging, and have a rollback plan. This guide has provided practical commands and examples to get you started, but the key is to develop a methodical mindset: measure, change, verify, and learn. As a next step, choose one subsystem (CPU, memory, I/O, or network) and establish a baseline using the tools described. Then, apply a single tuning parameter in a test environment and observe the effect. Over time, this practice will build your expertise and lead to a more stable and efficient Linux infrastructure.