Linux Security Hardening: A Practical, Step-by-Step Playbook

Intro

Linux security hardening is most effective when it follows an operational rhythm: observe before changing, apply the smallest justified change, verify the outcome, and prepare a clear recovery path. This playbook focuses on practical moves that developers, DevOps consultants, and startup teams can execute safely on common Linux distributions.

The examples below tie Linux hardening, access control, secrets, and permissions to concrete commands, expected signals, and reversibility. Related tools like systemd, Bash, and Docker are referenced only when they influence prerequisites, compatibility, or recovery.

Version and Environment Inventory

Before modifying anything, capture a read-only baseline. Knowing the OS family, kernel, init system, security framework (SELinux/AppArmor), and networking stack informs which commands and configs are valid.

Prerequisites

• Shell access with non-privileged user; escalate only when verification requires it.
• A second console/session you can fall back to during changes.

Read-only inventory (copy outputs into your change record with timestamps):

# OS and kernel
cat /etc/os-release
uname -r

# Init system and session context
ps -p 1 -o comm=
systemd-detect-virt --container || true

# Security framework presence
command -v getenforce >/dev/null && getenforce || echo "SELinux not present"
command -v aa-status >/dev/null && sudo aa-status || echo "AppArmor not present or needs sudo"

# Network listeners and firewall backend
ss -tulnp
command -v nft >/dev/null && nft list ruleset | head -n 20 || iptables -S | head -n 20

# SSH and sudo presence
pgrep -a sshd || systemctl status ssh || true
sudo -V | head -n 1

# Users and groups (read-only glance)
getent passwd | awk -F: '$3 >= 1000 {print $1":"$6}' | head
getent group | head

Interpretation tips

• If systemd is not PID 1 (e.g., in containers), some service commands differ; prefer process-level checks.
• SELinux/AppArmor state guides policy tuning. Enforcing/permissive/disabled changes risk and observability.
• Note inbound listeners and which firewall stack is active (nftables vs iptables). Plan accordingly.

Define expectations before changing anything

• Success signal: the precise command output or log line that proves the desired state.
• Failure signal: the exact symptom (e.g., connection refused, syntax error, AVC denial) that indicates rollback.

Safe Configuration Path

This section demonstrates small, auditable changes with explicit verification and rollback.

Example 1: Harden SSH access

Scope and blast radius

• Affects remote login. Risk: lockout. Keep an active root/privileged console or out-of-band access.

Steps

1. Observe current SSH config and access:

ss -tulnp | grep -E ':(22|<PORT>)'
sshd -T | sort | grep -E '^(permitrootlogin|passwordauthentication|allowusers)'
last -F | head -n 5

1. Backup and apply minimal restrictions:

sudo cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.bak.$(date +%Y%m%d%H%M%S)

# Edit with your editor; example directives below:
# - Disallow direct root login, prefer keys, limit user set
sudo sed -i 's/^#\?PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/^#\?PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
# Add (or adjust) to restrict to explicit accounts
sudo grep -q '^AllowUsers' /etc/ssh/sshd_config || echo 'AllowUsers <USER1> <USER2>' | sudo tee -a /etc/ssh/sshd_config

# Validate syntax before reload
sudo sshd -t
sudo systemctl reload sshd || sudo service ssh reload

Verification

• From a new terminal, test key-only login works and password auth fails:

ssh -o PreferredAuthentications=publickey <USER1>@<HOST>
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no <USER1>@<HOST>

• Expected: key auth succeeds; password-only attempt is rejected. Confirm logs:

journalctl -u ssh -n 50 --no-pager | tail

Recovery

• If lockout risk appears, revert quickly from the console:

sudo mv /etc/ssh/sshd_config.bak.* /etc/ssh/sshd_config
sudo systemctl restart sshd || sudo service ssh restart

Example 2: Least privilege for application secrets

Scope and blast radius

• Confines read access to a single service account. Low risk to system availability.

Assume a secret at /etc/myapp/secret.json consumed by service user <SERVICE_USER> in group myapp.

Steps

1. Observe current ownership and permissions:

namei -mo /etc/myapp/secret.json
stat -c '%U %G %a %n' /etc/myapp/secret.json
getfacl -p /etc/myapp/secret.json 2>/dev/null || true

1. Apply least privilege:

sudo chown root:myapp /etc/myapp/secret.json
sudo chmod 640 /etc/myapp/secret.json
# If <SERVICE_USER> is not in group myapp, grant explicit ACL read
sudo setfacl -m u:<SERVICE_USER>:r /etc/myapp/secret.json

Verification

namei -mo /etc/myapp/secret.json | tail -n 1  # should show root myapp 640
sudo -u <SERVICE_USER> bash -c 'test -r /etc/myapp/secret.json && echo OK'

Recovery

• Revert permissions and ACL if the service fails to start:

sudo setfacl -b /etc/myapp/secret.json   # clear ACLs
sudo chmod 644 /etc/myapp/secret.json

Example 3: Firewall baseline with ufw

Scope and blast radius

• Blocks unsolicited inbound traffic. High lockout risk if SSH is not explicitly allowed. Keep console access.

Prerequisites

• ufw installed but inactive, nftables/iptables backend available.

Steps

1. Observe current listeners and ufw state:

ss -tulnp | grep -v 127.0.0.1
sudo ufw status verbose || true

1. Define allowlist and default deny:

# Allow admin SSH from a fixed workstation
sudo ufw allow from <ADMIN_IP> to any port 22 proto tcp
# Allow required app ports (examples)
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
# Deny everything else inbound; keep outbound open
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw enable   # confirm when prompted

Verification

sudo ufw status numbered
nmap -Pn -p 22,80,443 <HOST>  # from allowed and non-allowed source, compare results

Recovery

• If you lose network access, from console:

sudo ufw disable
# As a last resort for nftables/iptables, carefully flush rules:
command -v nft >/dev/null && sudo nft flush ruleset || sudo iptables -F

Example 4: Network hardening via sysctl

Scope and blast radius

• Tightens network stack behavior. Low to moderate risk if values are conservative.

Steps

1. Observe current values:

sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.all.accept_redirects
sysctl net.ipv4.conf.all.send_redirects
sysctl net.ipv6.conf.all.accept_redirects 2>/dev/null || true

1. Persist sensible defaults in a dedicated file:

sudo tee /etc/sysctl.d/99-hardening.conf >/dev/null <<'EOF'
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
EOF
sudo systemctl restart systemd-sysctl || sudo sysctl --system

Verification

sysctl -a | grep -E 'rp_filter|accept_redirects|send_redirects' | sort

Recovery

• Remove or edit the file and reload:

sudo rm /etc/sysctl.d/99-hardening.conf
sudo systemctl restart systemd-sysctl || sudo sysctl --system

Verification and Diagnostics

Use simple checks to confirm security posture and catch regressions early.

• Access control
• last -F and journalctl -u ssh show who logged in and when.
• Test negative cases intentionally (e.g., password-only SSH) to confirm denials.

• Permissions and secrets
• namei -mo and getfacl reveal effective path permissions and ACLs.
• Use sudo -u <SERVICE_USER> to simulate service access without guessing.

• Firewall and exposure
• ss -tulnp and nmap from distinct networks validate the intended surface area.
• journalctl -k highlights kernel-level drops if logging is enabled by your firewall rules.

• Policy frameworks
• SELinux: ausearch -m AVC -ts recent and journalctl -t setroubleshoot (if present) surface denials.
• AppArmor: dmesg | grep DENIED or journalctl -k | grep apparmor show profile hits.

Define a standard “pass” for each change, store the command and its last known-good output, and set alerts on deviations where possible.

Failure Modes and Recovery

• Locked out of SSH
• Symptom: connection refused or key rejected post-change.
• Recovery: use console to restore /etc/ssh/sshd_config from backup and restart sshd. Keep at least one persistent privileged session open during changes.

• Broken sudoers
• Symptom: sudo fails with syntax error.
• Prevention: always edit with visudo, which validates before saving.
• Recovery: use root console to copy /etc/sudoers.bak back to /etc/sudoers and run visudo -c.

• Firewall cuts off management
• Symptom: cannot reach SSH or app ports from expected sources.
• Recovery: console run sudo ufw disable or carefully flush the ruleset. Re-apply allow rules first, then set default deny.

• Overzealous sysctl breaks networking
• Symptom: routing or connectivity regressions after reload.
• Recovery: remove the hardening file, reload systemd-sysctl or sysctl --system, and retest. Re-introduce settings incrementally.

• SELinux/AppArmor policy blocks services
• Symptom: service starts failing after policy or path changes.
• Recovery: check denials (ausearch or journalctl), adjust labels/profiles, or temporarily set SELinux permissive for diagnosis only. Re-enable enforcing after a targeted fix.

Operations Checklist

Use this repeatable flow for each hardening change:

1. Inventory

• Capture OS version, kernel, SELinux/AppArmor state, listeners, and active firewall backend.

1. Plan

• Define one smallest change, its success and failure signals, blast radius, and an explicit rollback command.

1. Prepare

• Create backups: cp -a <FILE> <FILE>.bak.$(date +%Y%m%d%H%M%S)
• Open a second console session. Confirm you have break-glass access.

1. Change

• Apply the modification. Validate syntax where supported (sshd -t, visudo -c, nft -c).

1. Verify

• Run the pre-declared verification commands. Test both allow and deny paths.

1. Observe

• Check logs (journalctl, dmesg) for silent failures or denials.

1. Document

• Record commands, outputs, timestamps, and final state for future audits and incident response.

1. Roll back if needed

• Use the prepared revert commands immediately when failure signals appear.

Conclusion

Effective Linux hardening is concrete, scoped, and reversible. Start with an inventory, make the smallest credible change, verify with explicit commands and logs, and keep a fast rollback nearby. The examples above for SSH, permissions, firewalling, and sysctl provide a template you can adapt to your distribution and deployment model. A disciplined workflow limits blast radius, protects secrets, and turns configuration from guesswork into a reliable, testable procedure.