Linux Production Operations Checklist with Practical Examples

Intro

Linux production operations can turn a simple change into an outage when the operator skips observation, version checking, or recovery planning. A checklist with practical examples helps teams move from an observed problem to a verified result without guessing.

This article is for developers, DevOps consultants, and technical startup teams who manage production Linux systems. It connects Linux operations, Linux checklist, Linux best practices, and Linux maintenance to concrete commands, expected output, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before touching a production system, you need an accurate inventory: what is installed, which version, how it is deployed, and what depends on it. A read-only inspection is the first step. Write the output to a log file with a timestamp so you can compare before and after states.

Example: Checking the operating system and kernel

Run these commands to capture the OS and kernel version:

cat /etc/os-release
uname -r

Expected output for a typical Ubuntu 22.04 LTS system:

PRETTY_NAME="Ubuntu 22.04.3 LTS"
VERSION_ID="22.04"
5.15.0-91-generic

If the kernel version is older than the recommended one for your workload, plan a maintenance window to update it. Record the current state in a file:

{
  echo "--- Inventory $(date -u +%Y-%m-%dT%H:%M:%SZ) ---"
  cat /etc/os-release
  uname -r
} >> /var/log/inventory.log

Use >> to append, never overwrite the log. If the expected version is absent, check the package manager history:

grep " install " /var/log/dpkg.log

This shows recent installations and can reveal unintended changes.

Check running services

Use systemctl to list active services and their states:

systemctl list-units --type=service --state=running

Look for services that should be running but are not, or services that are running unexpectedly. For example, if nginx.service is missing from the list, verify whether nginx is installed and enabled:

systemctl status nginx

Expected output for a running service:

● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-05-20 10:15:42 UTC; 1h 20min ago

If the service is inactive, start it with sudo systemctl start nginx, but first check the config with sudo nginx -t. This is a read-only syntax check:

sudo nginx -t

Expected success:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

If the syntax check fails, fix the configuration before starting the service to avoid a broken state.

Safe Configuration Path

Changing configuration is risky. Follow a safe path: back up the current config, make a scoped change, test it, and have a rollback plan. Always use version control or at least timestamped backups.

Example: Editing a systemd service file

Suppose you need to increase the open file limit for a service like PostgreSQL. First, locate the unit file:

systemctl cat postgresql.service

This shows the current unit file content. Copy it to a backup:

sudo cp /etc/systemd/system/postgresql.service /etc/systemd/system/postgresql.service.bak.$(date +%Y%m%d)

Now edit the file with sudo systemctl edit postgresql.service --full or use a text editor. The safe approach is to create an override file rather than modifying the original:

sudo systemctl edit postgresql.service

This opens a blank override file. Add:

[Service]
LimitNOFILE=65536

Save and exit. Then reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart postgresql.service

Verify the change took effect:

systemctl show postgresql.service -p LimitNOFILE

Expected output:

LimitNOFILE=65536

If the service fails to start, check the status and logs:

systemctl status postgresql.service
journalctl -u postgresql.service -n 50

If the problem is the recent change, roll back by removing the override:

sudo rm /etc/systemd/system/postgresql.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart postgresql.service

Always test configuration changes on a staging environment first if possible.

Example: Changing a kernel parameter with sysctl

To set a kernel parameter persistently, use /etc/sysctl.d/. For example, increase the maximum number of connections:

echo 'net.core.somaxconn = 1024' | sudo tee /etc/sysctl.d/99-custom.conf

Apply the change:

sudo sysctl -p /etc/sysctl.d/99-custom.conf

Verify:

sysctl net.core.somaxconn

Expected output:

net.core.somaxconn = 1024

To revert, remove the file and reload the default settings:

sudo rm /etc/sysctl.d/99-custom.conf
sudo sysctl --system

Verification and Diagnostics

After any change, verify that the system behaves as expected. This section covers commands to check processes, network sockets, disk usage, and logs.

Process verification

Use ps to inspect a specific process. For example, check the nginx master process:

ps -C nginx -o pid,user,cmd

Expected output:

  PID USER     CMD
 1234 root     nginx: master process /usr/sbin/nginx
 1235 www-data nginx: worker process

If the master process is missing, nginx is not running. You can also use pgrep:

pgrep -a nginx

Check listening ports

Use ss to list listening TCP ports:

ss -tlnp

Expected output includes lines like:

LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1234,fd=6))

This confirms that nginx is listening on port 80. If the expected port is missing, check the service configuration and logs.

Disk usage diagnostics

Check filesystem usage with df -h:

df -h

Example output:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        50G   30G   18G  63% /

If a filesystem is above 90% full, investigate with du:

sudo du -sh /var/log/*

This shows which directory is consuming the most space. Clean up old logs carefully, ensuring you do not delete active files.

Log analysis

Use journalctl to view recent logs for a service:

journalctl -u sshd.service -n 20 --no-pager

Look for error messages such as Authentication failure or Failed password. This can indicate a brute-force attack. You can also filter by time:

journalctl --since "2024-05-20 10:00:00" --until "2024-05-20 11:00:00"

Failure Modes and Recovery

Even careful operators encounter failures. This section describes common failure modes and recovery steps.

Service crash loop

If a service keeps crashing after start, use systemctl status to see the exit code and restart count:

systemctl status myapp.service

Look for lines like:

Active: activating (auto-restart) (Result: exit-code)

Check the logs for the root cause:

journalctl -u myapp.service -n 100

Common issues include missing configuration files, permission errors, or dependency failures. For example, if the app cannot connect to a database, you might see:

Error: connection refused to 127.0.0.1:5432

Fix the underlying issue (e.g., start the database service) and restart the app. If you cannot fix it immediately, you can temporarily stop the crash loop with:

sudo systemctl stop myapp.service

Then investigate without generating excessive logs.

Disk full recovery

When the root filesystem is full, the system may behave erratically. First, identify large files:

sudo find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null

Common culprits are old logs, core dumps, or package caches. For logs, you can truncate (not delete) a file to free space immediately:

sudo truncate -s 0 /var/log/syslog

For package caches, clean with:

sudo apt-get clean

After freeing space, verify with df -h.

Unresponsive system

If the system becomes unresponsive, try to regain control via SSH or console. If SSH is not responding, use a remote management interface (e.g., IPMI, cloud console). Check load average and memory:

uptime
free -h

If load is high, use top or htop to find the process consuming CPU:

top -bn1 | head -20

Identify the process and decide whether to kill it:

sudo kill -15 <PID>

Wait a few seconds; if it does not terminate, use kill -9:

sudo kill -9 <PID>

Always try graceful termination first to avoid data corruption.

Operations Checklist

Use this checklist before and after any change to a production Linux system.

Pre-change checklist

• [ ] Confirm the current state with read-only commands (e.g., systemctl status, df -h, ps aux).
• [ ] Record the output to a log file with a timestamp.
• [ ] Back up any configuration file you plan to modify.
• [ ] Verify the expected version of the software and dependencies.
• [ ] Identify the blast radius: which services or users are affected?
• [ ] Plan the rollback procedure.
• [ ] Schedule the change during a maintenance window if possible.
• [ ] Notify stakeholders.

Post-change checklist

• [ ] Verify the change took effect with the appropriate command.
• [ ] Check service health (e.g., systemctl status, application health endpoint).
• [ ] Review logs for any new errors.
• [ ] Monitor key metrics for a period (CPU, memory, disk, network).
• [ ] Document the change in your change log.
• [ ] Remove temporary backups only after confirming stability.

Conclusion

A Linux production operations checklist is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification for your Linux production environment. Record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as systemd, Bash, and Docker where relevant.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.