Linux Capacity Planning with Practical Examples: A Hands-On Guide

Intro

Capacity planning on Linux is about ensuring your systems have enough resources to handle current and future workloads without over-provisioning or under-provisioning. It is a continuous cycle of observing, measuring, and adjusting resources such as CPU, memory, disk I/O, and network bandwidth. This guide provides practical examples, commands, and strategies for developers, DevOps engineers, and technical startup teams who need to keep Linux systems reliable and cost-effective.

The goal is operational safety: observe before changing, limit the blast radius of any modification, use placeholders instead of real secrets, verify each change, and document recovery paths. By the end of this article, you will be able to inventory your environment, monitor key resources, configure safe limits, diagnose problems, recover from failures, and follow an operations checklist tailored to Linux capacity planning.

We will use standard Linux tools such as top, vmstat, iostat, free, df, ss, and systemd features, along with practical examples that you can adapt to your environment. All examples use placeholder values (e.g., your-app, your-service) that you should replace with your own names. Never use real credentials, tokens, or private keys in scripts or configuration files shown in articles or shared environments.

Version and Environment Inventory

Before making any capacity changes, you must know exactly what you are working with. Start by identifying the Linux distribution, kernel version, relevant services, and their current resource usage. This read-only observation sets a baseline and enables you to detect changes over time.

1. Identify the System and Kernel

Run these commands to get the distribution and kernel version:

cat /etc/os-release
uname -a

Example output snippet:

PRETTY_NAME="Ubuntu 22.04.3 LTS"
...
Linux your-host 5.15.0-91-generic #101-Ubuntu SMP ... x86_64 GNU/Linux

This tells you the OS version and kernel, which matter because some tuning parameters (like sysctl settings) and tools differ between kernel versions.

2. List Running Services and Resource Consumers

Identify which services are running and their resource footprint. Use systemctl to list active services:

systemctl list-units --type=service --state=running

To see the top processes by CPU and memory in real time, use top or ps:

top -b -n 1 | head -20

The output will show something like:

top - 14:32:11 up 12 days,  3:04,  2 users,  load average: 0.85, 0.70, 0.65
Tasks: 187 total,   1 running, 186 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.2 us,  0.6 sy,  0.0 ni, 97.9 id,  0.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  15886.7 total,   1024.5 free,   5102.3 used,   9759.9 buff/cache
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.  10123.4 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    871 mysql     20   0 2169124 451236  14208 S  12.5   2.8  123:45.67 mysqld

This snapshot shows load average, CPU idle, memory usage, and which process is consuming the most resources. In this example, mysqld is using 12.5% CPU and 2.8% memory, suggesting the database may need attention for capacity planning.

3. Capture Current State with Timestamps

Always record the current state with timestamps for future comparison. For example, create a baseline file:

date >> /var/log/capacity-baseline.txt
free -h >> /var/log/capacity-baseline.txt
vmstat 1 5 >> /var/log/capacity-baseline.txt

This appends a timestamp and memory/CPU statistics to a log file. Later, after changes, you can run the same commands and diff the results.

Prerequisites: Ensure you have read access to the system and the necessary permissions to run these commands (usually root or sudo for some files). Blast radius: These are read-only commands; they do not alter system state. Expected result: You should see a list of services and resource usage that matches what you expect. Failure signal: If a service is missing from the list or resource usage is unexpectedly high, investigate before proceeding.

Safe Configuration Path

When you need to adjust capacity, follow a safe path: make small, reversible changes with verification at each step. Below are common areas where you can safely fine-tune Linux capacity.

1. Adjust System Limits with sysctl

Linux kernel parameters can be tuned at runtime using sysctl. For example, to increase the maximum number of file descriptors (useful for busy web servers), you can change fs.file-max.

First, check the current value:

sysctl fs.file-max

Output:

fs.file-max = 9223372036854775807

This is the system-wide limit. To increase it (if needed), edit /etc/sysctl.conf or a file in /etc/sysctl.d/ and add or modify:

fs.file-max = 2097152

Then apply the change:

sudo sysctl -p

Verify:

sysctl fs.file-max

Expected output: fs.file-max = 2097152.

Blast radius: This change affects the whole system but is generally safe if you do not set an extremely low value. Recovery: To revert, restore the original value and run sysctl -p again.

2. Configure User and Process Limits with ulimit

For per-user or per-process resource limits, use ulimit or configure limits.conf. For example, to set a soft limit on the number of open files for a user, add to /etc/security/limits.conf:

your-user soft nofile 65536
your-user hard nofile 65536

Have the user log out and back in, then check:

ulimit -n

Expected: 65536.

Prerequisites: You need root to edit limits.conf. Blast radius: Affects only the specified user's new sessions. Recovery: Remove or adjust the lines and have the user log in again.

3. Use systemd Resource Controls

If your services are managed by systemd, you can set resource limits directly in the unit file or via systemctl set-property. For example, to limit memory usage for a service named your-service:

sudo systemctl set-property your-service MemoryMax=1G

This applies immediately and persists across reboots (it creates a drop-in file). Verify with:

systemctl show your-service -p MemoryMax

Expected output: MemoryMax=1073741824 (1 GiB in bytes).

To revert, run:

sudo systemctl set-property your-service MemoryMax=infinity

Or remove the drop-in file from /etc/systemd/system.control/.

Blast radius: Only affects the specified service. Failure signal: If the service tries to exceed the limit, it may be killed by the OOM killer; monitor logs and adjust accordingly.

4. Adjust Swappiness

Swappiness controls how aggressively the kernel swaps memory pages. For systems with enough RAM, you might lower swappiness to reduce unnecessary disk I/O. Check current value:

cat /proc/sys/vm/swappiness

Default is usually 60. To change to 10, edit /etc/sysctl.conf:

vm.swappiness = 10

Apply and verify as before. Blast radius: Affects memory management system-wide but is safe for most workloads. Recovery: Set back to original value (e.g., 60) and reapply.

Verification and Diagnostics

After making changes or when problems arise, you need to verify that capacity is sufficient and diagnose bottlenecks. Here are key areas and commands.

1. CPU Monitoring and Load Average

Load average indicates the number of processes waiting for CPU. Use uptime or top:

uptime

Output:

 14:45:00 up 12 days,  3:17,  2 users,  load average: 1.10, 0.95, 0.80

Interpretation: For a single-core system, load average should be below 1.00 for no waiting; for a multi-core system, divide by the number of cores. If load average consistently exceeds the number of CPU cores, investigate which processes are consuming CPU with top or pidstat:

pidstat 1 5

This shows per-process CPU usage over time, helping identify the culprit.

2. Memory Usage and OOM

Check memory with free -h:

free -h

Example:

              total        used        free      shared  buff/cache   available
Mem:           15Gi       5.0Gi       1.0Gi        50Mi       9.0Gi        9.5Gi
Swap:         2.0Gi          0B       2.0Gi

Focus on the available column, which estimates memory for new applications. If available is low, check which processes use memory with ps aux --sort=-%mem | head. Also monitor for OOM (Out Of Memory) kills in the kernel log:

dmesg | grep -i "out of memory"

If OOM kills occur, consider increasing RAM, adding swap, or tuning application memory.

3. Disk I/O and Storage Capacity

Use iostat for I/O statistics (install sysstat if not present):

iostat -xz 1 5

Look at %util and await for each device. High %util (near 100%) and high await indicate disk saturation. For storage capacity, use df -h:

df -h

Example:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   40G   10G  80% /

If a filesystem is over 80% full, plan for cleanup or expansion.

4. Network Capacity

Check network interface throughput with sar -n DEV (from sysstat) or iftop. For a quick view, use:

sar -n DEV 1 5

Output shows rxkB/s and txkB/s per interface. Compare with the interface's maximum bandwidth (e.g., 1 Gbps = ~125 MB/s). If near saturation, consider load balancing or upgrading.

Failure Modes and Recovery

Even with careful planning, failures happen. Here are common capacity-related failures and recovery steps.

1. High CPU Load and Slow Response

Failure: Load average spikes above CPU cores, processes become unresponsive. Diagnosis: Use top to find the offending process. Immediate mitigation: If it's a runaway process, you can temporarily reduce its priority with renice:

sudo renice +10 -p <PID>

Or, if it's a service, restart it:

sudo systemctl restart your-service

Long-term recovery: Identify the cause (e.g., code bug, insufficient CPU), fix it, and consider adding CPU resources or optimizing the application.

2. Out of Memory (OOM) Kills

Failure: Applications get killed by the OOM killer. Diagnosis: Check logs:

dmesg | grep -i oom
journalctl -k | grep -i oom

Immediate mitigation: If a service was killed, restart it. To prevent immediate recurrence, temporarily reduce memory usage by stopping non-critical services or increasing swap:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Long-term recovery: Tune application memory, adjust systemd memory limits, or add physical RAM.

3. Disk Full

Failure: No space left on device. Diagnosis: df -h shows 100% usage. Find large files:

sudo du -ah / | sort -rh | head -20

Immediate mitigation: Delete or compress large log files, clear package cache (sudo apt clean or sudo yum clean all). Long-term recovery: Implement log rotation (logrotate), monitor disk usage, and expand storage if necessary.

4. Network Saturation

Failure: High packet loss or latency, interface at max throughput. Diagnosis: Use sar -n DEV to see throughput; use ping and traceroute to test. Immediate mitigation: Limit bandwidth of non-critical services using tc (traffic control) or pause heavy jobs. Long-term recovery: Upgrade network hardware, use load balancers, or optimize data transfer.

For all failures, document the incident, the steps taken, and the recovery verification. This ensures you can handle similar issues faster in the future.

Operations Checklist

Use this checklist for every capacity planning task on Linux to ensure safety and effectiveness.

This checklist can be adapted and automated. For example, you can create a script that gathers all monitoring data and alerts on failures.

Conclusion

Linux capacity planning is not a one-time task but an ongoing practice. By inventorying your environment, monitoring key resources, safely adjusting limits, diagnosing problems, and preparing for failures, you can keep your systems running smoothly and cost-effectively.

Remember to always observe before changing, use read-only commands for baselines, protect sensitive data with placeholders, limit the blast radius of any change, verify results with concrete commands, and document recovery paths.

Start with one low-risk verification on a test system: capture the current CPU and memory baseline, monitor it for a day, and then identify if any resource is nearing its limits. Use the operations checklist in this article to guide your process.

With these practical examples and a disciplined approach, you will be well-equipped to handle capacity planning on Linux systems.