systemd Capacity Planning: Practical Examples for Resource Management

Intro

systemd is the default init system on most modern Linux distributions, and it does far more than start and stop services. It also provides a unified interface for managing service resources: CPU, memory, I/O, tasks, and more. Capacity planning in systemd is the practice of understanding how much of each resource a service needs, setting appropriate limits, and monitoring usage to prevent one service from starving others or crashing the host.

This guide is for developers, DevOps consultants, and technical startup teams who need to keep production services stable without over-provisioning hardware. It covers how to inspect current consumption, set resource controls via unit files or commands, verify that limits are applied, diagnose common failures, and recover when something goes wrong.

The approach throughout is operational safety first: always observe before changing, make one scoped change at a time, use placeholders instead of real secrets, verify the result immediately, and document a rollback path.

Version and Environment Inventory

Before touching any limits, know exactly what you are working with. Start with the systemd version and basic host facts.

systemctl --version

Expected output includes lines like:

systemd 252 (252.5-1ubuntu1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GSS +CRYPTO +GNUTLS +SYNTHETIC_ERRNO +WATCH_QUEUE

Also check the kernel version, because some resource controllers depend on kernel support:

uname -r

A typical response on Ubuntu 22.04 is 5.15.0-91-generic. If you know you need a specific cgroup version or controller, verify it early. For example, check the cgroup version in use:

stat -fc %T /sys/fs/cgroup/

On a system with cgroup v2, it prints cgroup2fs. On cgroup v1, it prints tmpfs.

For each service you plan to tune, record its currently active unit file and any drop-ins. For a service named webapp:

systemctl cat webapp

The output shows the main unit file and any override files under /etc/systemd/system/webapp.service.d/. That is where you will often add limits; knowing whether a drop-in already exists avoids clobbering prior work.

Capture the current resource usage snapshot as a baseline. Use systemctl show to see the effective properties, including memory and CPU accounting if enabled. Enable accounting first if needed with systemctl set-property or by editing the unit, but note that it may be off by default on some distributions.

systemctl show webapp -p MemoryCurrent -p CPUUsageNSec

Sample output:

MemoryCurrent=18446744073709551615
CPUUsageNSec=18446744073709551615

The value 18446744073709551615 is the maximum 64-bit unsigned integer, indicating that accounting is not enabled. After enabling MemoryAccounting=yes and CPUAccounting=yes in the unit and reloading, the same command might return:

MemoryCurrent=47923200
CPUUsageNSec=124000000

That means the service currently uses about 45.7 MiB of memory and has consumed 0.124 seconds of CPU time so far.

Document this baseline before any changes. Write down the date, the exact command output, and any relevant load conditions (e.g., "during peak traffic with 200 concurrent users"). This gives you a reference for later comparisons.

Safe Configuration Path

Once you know the current state, decide what limit to impose. The safest path is to use systemctl set-property at runtime first, because it does not require restarting the service and can be reverted instantly. However, runtime changes are lost on reboot, so once you confirm the value works, persist it in a drop-in unit file.

Runtime (non-persistent) change

For a service named webapp, limit its memory to 512 MiB and CPU to 1.5 cores:

sudo systemctl set-property webapp MemoryMax=512M CPUQuota=150%

The CPUQuota percentage is relative to a single core: 150% means 1.5 cores of CPU time. After running this, verify the property:

systemctl show webapp -p MemoryMax -p CPUQuota

Expected output:

MemoryMax=536870912
CPUQuota=150%

Note that MemoryMax is displayed in bytes. The service is now capped at 512 MiB of memory and 1.5 CPU cores.

Persistent change via drop-in

To make the change permanent, create a drop-in file. Start by making the directory if it does not exist:

sudo mkdir -p /etc/systemd/system/webapp.service.d/

Then create a file named limits.conf inside that directory:

[Service]
MemoryMax=512M
CPUQuota=150%
TasksMax=256

Equivalently, you can run:

sudo systemctl edit webapp

This opens an editor for a new override file. Add the same three lines, save, and exit. Then reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart webapp

After restart, verify again with systemctl show webapp -p MemoryMax -p CPUQuota -p TasksMax. The limits should now be active and survive reboots.

Choosing limit values

The hardest part of capacity planning is selecting numbers that prevent resource exhaustion without starving the service. A good starting point is to run the service under realistic load and observe its peak usage. Suppose you can trigger a load test that mimics your busiest hour. With accounting enabled, record the peak memory and CPU over several runs. If you see peak memory around 380 MiB, setting MemoryMax=512M gives a comfortable headroom. If peak CPU is 1.2 cores, CPUQuota=150% leaves room for bursts.

Always set limits above observed peaks by a safety margin, and monitor for throttling or OOM kills under the same load after applying limits.

Verification and Diagnostics

After applying any limit, verify not only that the property is set, but that it is being enforced. systemd exposes current usage and throttling counters.

Check current memory usage and limit

systemctl show webapp -p MemoryCurrent -p MemoryMax

Example:

MemoryCurrent=234881024
MemoryMax=536870912

Here the service uses about 224 MiB, below the 512 MiB cap. If MemoryCurrent equals or approaches MemoryMax, the service may be about to hit the limit.

Check CPU throttling

When a service exceeds its CPUQuota, the kernel throttles it. systemd tracks the number of throttle events and total throttled time.

systemctl show webapp -p CPUThrottling

Possible output:

CPUThrottling=yes

If throttling has occurred, the following properties are non-zero:

systemctl show webapp -p CPUUsageNSec -p CPUTimeNSec -p CPUThrottledNSec -p CPUThrottledUSec

Sample:

CPUUsageNSec=1240000000
CPUTimeNSec=1500000000
CPUThrottledNSec=260000000
CPUThrottledUSec=260000

That means the service would have used 1.5 seconds of CPU, but 0.26 seconds were throttled away.

Check for OOM kills caused by MemoryMax

If a service tries to allocate more memory than MemoryMax, the kernel's OOM killer may kill a process inside the cgroup. Check the journal:

journalctl -u webapp --since "10 minutes ago" | grep -i "out of memory"

Or look at the cgroup's memory.events file. This path depends on the cgroup version and hierarchy. On cgroup v2 under systemd, you can find the cgroup path with:

systemctl show webapp -p ControlGroup

Then examine /sys/fs/cgroup/<path>/memory.events. The file reports oom and oom_kill counts. A non-zero oom_kill means the limit was too low.

Failure Modes and Recovery

Capacity limits can cause subtle failures. The most common are service crashes from memory limits, unexplained slowness from CPU throttling, and failed starts from invalid unit file syntax. Know how to recognize each and recover quickly.

Service killed by memory limit

Symptom: The service restarts frequently, and systemctl status webapp shows Main process exited, code=killed, status=9/KILL. Sometimes the status shows status=137/SIGKILL or an OOM message.

Diagnosis: Check journalctl -u webapp -e for lines containing Memory cgroup out of memory or oom-kill.

Recovery: Raise MemoryMax temporarily to unblock the service, then investigate the actual memory usage pattern. For a quick runtime increase:

sudo systemctl set-property webapp MemoryMax=1G

Then decide whether to make it permanent and whether the application has a memory leak that needs a code fix.

CPU throttling causing high latency

Symptom: Application response times increase dramatically under load, but the host CPU is not saturated. The service may show CPUThrottling=yes.

Diagnosis: Check systemctl show webapp -p CPUThrottledNSec over time. If the throttled time grows during load, the quota is too low.

Recovery: Increase CPUQuota in small steps. For example, raise from 150% to 200%:

sudo systemctl set-property webapp CPUQuota=200%

Observe latency and throttling metrics for a few minutes. If latency recovers and throttling drops to zero, you have found the issue. If not, consider other bottlenecks.

Unit file syntax error prevents start

Symptom: After editing a drop-in and running systemctl restart webapp, the service fails to start, and systemctl status webapp shows an error like Failed to parse resource value, ignoring: abc or Invalid argument.

Diagnosis: Run systemd-analyze verify /etc/systemd/system/webapp.service.d/limits.conf to check for syntax errors. Or run systemctl daemon-reload, which often prints the offending line.

Recovery: Comment out or remove the problematic line in the drop-in, save, and run systemctl daemon-reload again. Once the service starts, correct the value. Always test changes in a staging environment when possible.

Forgetting to reload after unit edit

Symptom: You edit a unit file, but the service still shows old property values after restart.

Diagnosis: Compare systemctl cat webapp with systemctl show webapp -p MemoryMax. If they differ, systemd has not reloaded the unit definitions.

Recovery: Run sudo systemctl daemon-reload, then restart the service again. Make it a habit to run daemon-reload after every unit file change.

Common Pitfalls

Even experienced operators make mistakes. Here are frequent pitfalls in systemd capacity planning and how to avoid them.

Setting limits based on average usage instead of peak

Why it happens: Monitoring dashboards often show average CPU and memory over long periods, which hides spikes. A service that averages 200 MiB may peak at 800 MiB during a daily batch job.

How to avoid: Collect peak metrics using tools like cAdvisor, systemd-cgtop, or Prometheus exporters with cgroup metrics. Set limits above the 95th or 99th percentile of observed peaks, and include a safety margin.

Not enabling accounting before relying on metrics

Why it happens: Many tutorials show systemctl show output with current usage, but on some distributions, memory and CPU accounting are disabled by default. The service appears to use 18 exabytes, which is the sentinel value for "not accounted."

How to avoid: Explicitly set MemoryAccounting=yes, CPUAccounting=yes, and TasksAccounting=yes in a drop-in for every service you need to measure. Reload and restart, then verify that MemoryCurrent is a reasonable number.

Applying runtime changes and assuming they persist

Why it happens: systemctl set-property is fast and reversible, so it is tempting to use it in production and forget about it. After a reboot, the service reverts to the original unit file.

How to avoid: Treat runtime changes as temporary. After confirming a value works, immediately create a permanent drop-in with the same settings and test a restart.

Using absolute constraints without headroom

Why it happens: Teams want to be strict to prevent runaway services, but setting MemoryMax exactly at the observed peak leaves no room for small variations. The service gets OOM-killed intermittently.

How to avoid: Add at least 20-30% headroom over the observed peak for memory. For CPU, allow for bursts by setting the quota higher than average but below the host's total cores.

Ignoring cgroup version differences

Why it happens: Some resource controls, such as MemorySwapMax, behave differently or are unavailable on cgroup v2. Older guides may refer to cgroup v1 paths.

How to avoid: Check the cgroup version before following a tutorial. Use systemd-provided interfaces (systemctl commands and unit file directives) instead of manually writing to cgroup files, because systemd abstracts many differences.

Operations Checklist

Use this checklist before and after making capacity changes. Each item names a specific owner (an individual, not a group) and a review frequency.

• [ ] Baseline captured: Record memory, CPU, and task counts under typical load. Owner: Priya Shah, Engineering Lead. Review quarterly or whenever the service load profile changes.
• [ ] Accounting enabled: Verify MemoryAccounting=yes and CPUAccounting=yes in the unit. Owner: Alex Chen, DevOps Engineer. Review after any OS upgrade or image rebuild.
• [ ] Target limits defined with headroom: e.g., MemoryMax=512M, CPUQuota=150% based on measured peak plus 25% margin. Owner: Priya Shah. Review monthly.
• [ ] Runtime test performed: Use systemctl set-property and observe for at least one full load cycle. Owner: Alex Chen. Review before every persistent rollout.
• [ ] Drop-in file created and reviewed: File placed in /etc/systemd/system/<service>.service.d/limits.conf. Owner: Alex Chen. Review immediately after creation.
• [ ] daemon-reload and restart completed: Service restarted without errors. Owner: Alex Chen. Verify in every change window.
• [ ] Post-change verification: Run systemctl show and journalctl checks; confirm no OOM kills or throttling. Owner: Priya Shah. Review after every change.
• [ ] Rollback plan documented: Commands to revert the drop-in and revert to previous limits. Owner: Priya Shah. Review before each change.
• [ ] Monitoring alert set: Alert when service memory usage exceeds 85% of MemoryMax or when CPUThrottling=yes for more than 5 minutes. Owner: Alex Chen. Review monthly.

Conclusion

systemd capacity planning is an ongoing operational practice, not a one-time configuration task. By measuring current usage, applying limits through a safe path, verifying enforcement, and preparing for common failures, you can keep services stable and prevent resource contention on shared hosts.

The techniques here apply to any systemd-managed service, whether it is a custom web application, a database, or a background worker. Start with one low-risk limit on a non-critical service, follow the checklist, and build confidence before rolling out to production.

Remember that every limit is a constraint: too high and it does not protect anything, too low and it breaks the service. Regular review of metrics and limits, with clear ownership, ensures that your capacity planning keeps up with changing workloads.