## Intro

Linux production operations can turn a simple change into an outage when the operator skips observation, version checking, or recovery planning. A checklist with practical examples helps teams move from an observed problem to a verified result without guessing.

This article is for developers, DevOps consultants, and technical startup teams who manage production Linux systems. It connects Linux operations, Linux checklist, Linux best practices, and Linux maintenance to concrete commands, expected output, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

## Version and Environment Inventory

Before touching a production system, you need an accurate inventory: what is installed, which version, how it is deployed, and what depends on it. A read-only inspection is the first step. Write the output to a log file with a timestamp so you can compare before and after states.

### Example: Checking the operating system and kernel

Run these commands to capture the OS and kernel version:

cat /etc/os-release
uname -r 
 Expected output for a typical Ubuntu 22.04 LTS system:

PRETTY_NAME="Ubuntu 22.04.3 LTS"
VERSION_ID="22.04"
5.15.0-91-generic 
 If the kernel version is older than the recommended one for your workload, plan a maintenance window to update it. Record the current state in a file:

{
 echo "--- Inventory $(date -u +%Y-%m-%dT%H:%M:%SZ) ---"
 cat /etc/os-release
 uname -r
} >> /var/log/inventory.log 
 Use >> to append, never overwrite the log. If the expected version is absent, check the package manager history:

grep " install " /var/log/dpkg.log 
 This shows recent installations and can reveal unintended changes.

### Check running services

Use systemctl to list active services and their states:

systemctl list-units --type=service --state=running 
 Look for services that should be running but are not, or services that are running unexpectedly. For example, if nginx.service is missing from the list, verify whether nginx is installed and enabled:

systemctl status nginx 
 Expected output for a running service:

● nginx.service - A high performance web server and a reverse proxy server
 Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
 Active: active (running) since Mon 2024-05-20 10:15:42 UTC; 1h 20min ago 
 If the service is inactive, start it with sudo systemctl start nginx , but first check the config with sudo nginx -t . This is a read-only syntax check:

sudo nginx -t 
 Expected success:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful 
 If the syntax check fails, fix the configuration before starting the service to avoid a broken state.

## Safe Configuration Path

Changing configuration is risky. Follow a safe path: back up the current config, make a scoped change, test it, and have a rollback plan. Always use version control or at least timestamped backups.

### Example: Editing a systemd service file

Suppose you need to increase the open file limit for a service like PostgreSQL. First, locate the unit file:

systemctl cat postgresql.service 
 This shows the current unit file content. Copy it to a backup:

sudo cp /etc/systemd/system/postgresql.service /etc/systemd/system/postgresql.service.bak.$(date +%Y%m%d) 
 Now edit the file with sudo systemctl edit postgresql.service --full or use a text editor. The safe approach is to create an override file rather than modifying the original:

sudo systemctl edit postgresql.service 
 This opens a blank override file. Add:

[Service]
LimitNOFILE=65536 
 Save and exit. Then reload systemd and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart postgresql.service 
 Verify the change took effect:

systemctl show postgresql.service -p LimitNOFILE 
 Expected output:

LimitNOFILE=65536 
 If the service fails to start, check the status and logs:

systemctl status postgresql.service
journalctl -u postgresql.service -n 50 
 If the problem is the recent change, roll back by removing the override:

sudo rm /etc/systemd/system/postgresql.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart postgresql.service 
 Always test configuration changes on a staging environment first if possible.

### Example: Changing a kernel parameter with sysctl

To set a kernel parameter persistently, use /etc/sysctl.d/ . For example, increase the maximum number of connections:

echo 'net.core.somaxconn = 1024' | sudo tee /etc/sysctl.d/99-custom.conf 
 Apply the change:

sudo sysctl -p /etc/sysctl.d/99-custom.conf 
 Verify:

sysctl net.core.somaxconn 
 Expected output:

net.core.somaxconn = 1024 
 To revert, remove the file and reload the default settings:

sudo rm /etc/sysctl.d/99-custom.conf
sudo sysctl --system 

## Verification and Diagnostics

 After any change, verify that the system behaves as expected. This section covers commands to check processes, network sockets, disk usage, and logs.

### Process verification

Use ps to inspect a specific process. For example, check the nginx master process:

ps -C nginx -o pid,user,cmd 
 Expected output:

PID USER CMD
 1234 root nginx: master process /usr/sbin/nginx
 1235 www-data nginx: worker process 
 If the master process is missing, nginx is not running. You can also use pgrep :

pgrep -a nginx 

### Check listening ports

 Use ss to list listening TCP ports:

ss -tlnp 
 Expected output includes lines like:

LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1234,fd=6)) 
 This confirms that nginx is listening on port 80. If the expected port is missing, check the service configuration and logs.

### Disk usage diagnostics

Check filesystem usage with df -h :

df -h 
 Example output:

Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 30G 18G 63% / 
 If a filesystem is above 90% full, investigate with du :

sudo du -sh /var/log/* 
 This shows which directory is consuming the most space. Clean up old logs carefully, ensuring you do not delete active files.

### Log analysis

Use journalctl to view recent logs for a service:

journalctl -u sshd.service -n 20 --no-pager 
 Look for error messages such as Authentication failure or Failed password . This can indicate a brute-force attack. You can also filter by time:

journalctl --since "2024-05-20 10:00:00" --until "2024-05-20 11:00:00" 

## Failure Modes and Recovery

 Even careful operators encounter failures. This section describes common failure modes and recovery steps.

### Service crash loop

If a service keeps crashing after start, use systemctl status to see the exit code and restart count:

systemctl status myapp.service 
 Look for lines like:

Active: activating (auto-restart) (Result: exit-code) 
 Check the logs for the root cause:

journalctl -u myapp.service -n 100 
 Common issues include missing configuration files, permission errors, or dependency failures. For example, if the app cannot connect to a database, you might see:

Error: connection refused to 127.0.0.1:5432 
 Fix the underlying issue (e.g., start the database service) and restart the app. If you cannot fix it immediately, you can temporarily stop the crash loop with:

sudo systemctl stop myapp.service 
 Then investigate without generating excessive logs.

### Disk full recovery

When the root filesystem is full, the system may behave erratically. First, identify large files:

sudo find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null 
 Common culprits are old logs, core dumps, or package caches. For logs, you can truncate (not delete) a file to free space immediately:

sudo truncate -s 0 /var/log/syslog 
 For package caches, clean with:

sudo apt-get clean 
 After freeing space, verify with df -h .

### Unresponsive system

If the system becomes unresponsive, try to regain control via SSH or console. If SSH is not responding, use a remote management interface (e.g., IPMI, cloud console). Check load average and memory:

uptime
free -h 
 If load is high, use top or htop to find the process consuming CPU:

top -bn1 | head -20 
 Identify the process and decide whether to kill it:

sudo kill -15 <PID> 
 Wait a few seconds; if it does not terminate, use kill -9 :

sudo kill -9 <PID> 
 Always try graceful termination first to avoid data corruption.

## Operations Checklist

Use this checklist before and after any change to a production Linux system.

### Pre-change checklist

- [ ] Confirm the current state with read-only commands (e.g., systemctl status , df -h , ps aux ).

- [ ] Record the output to a log file with a timestamp.

- [ ] Back up any configuration file you plan to modify.

- [ ] Verify the expected version of the software and dependencies.

- [ ] Identify the blast radius: which services or users are affected?

- [ ] Plan the rollback procedure.

- [ ] Schedule the change during a maintenance window if possible.

- [ ] Notify stakeholders.

### Post-change checklist

- [ ] Verify the change took effect with the appropriate command.

- [ ] Check service health (e.g., systemctl status , application health endpoint).

- [ ] Review logs for any new errors.

- [ ] Monitor key metrics for a period (CPU, memory, disk, network).

- [ ] Document the change in your change log.

- [ ] Remove temporary backups only after confirming stability.

## Conclusion

A Linux production operations checklist is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification for your Linux production environment. Record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as systemd, Bash, and Docker where relevant.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.