Linux filesystem hierarchy explained with practical examples: practical implementation guide

Intro

The Linux filesystem hierarchy is a contract between OS, tools, and humans. When you put files in the right place, you get predictable installs, simpler backups, easier debugging, and smoother handoffs across teams.

Use three questions to decide where something belongs:

• Is it static or variable? Static code and assets go under /usr (or /usr/local). Variable data and logs go under /var.
• Is it for the system or for a user? System-wide items live under /, /usr, /etc, /var. User files and settings live under /home/USER.
• Is it ephemeral or persistent? Ephemeral runtime files go in /run or /tmp. Persistent state goes under /var/lib.

This guide explains the major directories, shows you where common application files belong, and walks through a minimal, safe setup you can run locally.

Workflow Overview

What lives where (practical mapping)

• / (root): The top of the tree. Do not scatter app files directly here.
• /bin, /sbin: Essential commands for booting and repair. On modern systems, these may be links into /usr.
• /usr: Read-only app code and shared data managed by the OS package manager.
• /usr/bin: User executables.
• /usr/sbin: System executables.
• /usr/lib or /usr/lib64: Libraries.
• /usr/share: Architecture-independent data (icons, locales, man pages).
• /usr/local: Admin-installed software that is not managed by the OS package manager. Mirrors /usr layout: /usr/local/bin, /usr/local/lib, etc.
• /opt: Self-contained, vendor-supplied apps. Good for a bundled runtime or a single product tree like /opt/acme/hello-app.
• /etc: System-wide configuration (static or edited by admins). No app should write here at runtime.
• /var: Variable data written at runtime.
• /var/lib/APP: Durable app state (databases, metadata).
• /var/log/APP: Log files.
• /var/cache/APP: Rebuildable caches.
• /var/spool/APP: Queues and incoming work.
• /run: Volatile runtime files (tmpfs). PIDs, locks, sockets, and short-lived control files, e.g. /run/APP/APP.pid.
• /tmp: Temporary scratch space, often cleaned automatically. Never rely on persistence or security here.
• /home/USER: User data and dotfiles. Not for system services.
• /root: Home directory for the root user.
• /mnt: Admin-mounted filesystems for temporary or manual use.
• /media: Automounted removable media (USB, CD).
• /srv: Data served by services (e.g., web content) when you want a conventional service root.
• /dev: Device nodes.
• /proc and /sys: Virtual filesystems that expose kernel and device info.
• /boot: Kernel, initramfs, and bootloader files.

Common application file placement

• Binaries:
• From distro packages: /usr/bin/yourapp (plus libs under /usr/lib, data under /usr/share).
• Locally installed by admins: /usr/local/bin/yourapp.
• Vendor bundle: /opt/vendor/yourapp/bin/yourapp and friends under the same tree.
• Configuration (system-wide): /etc/yourapp/
• Example: /etc/yourapp/config.yml and /etc/yourapp/env.conf
• Persistent state: /var/lib/yourapp/
• Example: /var/lib/yourapp/state.db
• Logs: /var/log/yourapp/
• Example: /var/log/yourapp/app.log
• Caches: /var/cache/yourapp/
• Queues: /var/spool/yourapp/
• Runtime (volatile): /run/yourapp/
• Example: /run/yourapp/yourapp.pid, /run/yourapp/yourapp.sock
• Temp files: /tmp/yourapp-XXXXXX
• Mount points for data volumes: /mnt/data or a subdir specific to your team, mounted via fstab or systemd.

Permissions and ownership quick rules

• Create a dedicated system user and group for your service. Avoid running as root.
• Own runtime and state directories by that user: group. Use least privileges.
• Typical modes:
• Config: 640 or 600 when secrets are present.
• State and logs: 750 for dirs, 640 for files.
• Runtime dirs: 750 for dirs; PID and socket files created by the service.

A short checklist for new services

1. Choose install target: /usr (packaged), /usr/local (admin-managed), or /opt (vendor bundle).
2. Place configs in /etc/APP.
3. Create /var/lib/APP, /var/log/APP, /var/cache/APP, /var/spool/APP as needed.
4. Create /run/APP at service start (systemd can manage it) or ensure the app creates it.
5. Set correct ownership and modes.
6. Provide a service unit (e.g., systemd) and ensure logs are discoverable.
7. Document paths in README and backup scripts.

Local Pilot Plan

Keep the first implementation narrow, measurable, and easy to inspect locally. Below is a small, safe example using a dummy service named hello-app. Adjust names to match your stack.

1) Create a service user

sudo useradd --system --no-create-home --shell /usr/sbin/nologin hello

2) Install the binary

Pick one style and stick to it.

Option A: admin-managed under /usr/local

sudo install -m 0755 ./hello-app /usr/local/bin/hello-app

Option B: vendor bundle under /opt

sudo mkdir -p /opt/acme/hello-app/bin
sudo install -m 0755 ./hello-app /opt/acme/hello-app/bin/hello-app
# Optional convenience symlink
sudo ln -sf /opt/acme/hello-app/bin/hello-app /usr/local/bin/hello-app

3) Create configuration

sudo mkdir -p /etc/hello-app
printf '%s\n' \
  'listen_addr=127.0.0.1:8080' \
  'log_level=info' \
  'data_dir=/var/lib/hello-app' | sudo tee /etc/hello-app/hello.conf >/dev/null
sudo chmod 640 /etc/hello-app/hello.conf
sudo chown root: hello /etc/hello-app/hello.conf

4) Create state, log, cache, and runtime dirs

sudo mkdir -p /var/lib/hello-app /var/log/hello-app /var/cache/hello-app /var/spool/hello-app
sudo chown -R hello: hello /var/lib/hello-app /var/log/hello-app /var/cache/hello-app /var/spool/hello-app
sudo chmod 750 /var/lib/hello-app /var/log/hello-app /var/cache/hello-app /var/spool/hello-app

# Let the service manager create /run/hello-app, or pre-create for testing
sudo mkdir -p /run/hello-app
sudo chown hello: hello /run/hello-app
sudo chmod 750 /run/hello-app

5) Add a minimal systemd unit (if your distro uses systemd)

# /etc/systemd/system/hello-app.service
[Unit]
Description=Hello App service
After=network.target

[Service]
User=hello
Group=hello
EnvironmentFile=-/etc/hello-app/hello.conf
ExecStart=/usr/local/bin/hello-app --config /etc/hello-app/hello.conf \\
  --data-dir ${data_dir:-/var/lib/hello-app} --listen ${listen_addr:-127.0.0.1:8080}
WorkingDirectory=/
RuntimeDirectory=hello-app
RuntimeDirectoryMode=0750
Restart=on-failure

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now hello-app
sudo systemctl status hello-app --no-pager

6) Validate behavior

• Binary is on PATH:

  command -v hello-app

• Service is listening locally:

  ss -lntp | grep 8080 || sudo lsof -nP -iTCP:8080 -sTCP:LISTEN

• Logs are where you expect:

  ls -l /var/log/hello-app
  sudo journalctl -u hello-app --no-pager | tail -n 50

• State directory changes on use:

  ls -l /var/lib/hello-app

• Runtime files exist while running:

  ls -l /run/hello-app

7) Add a data volume (optional, local)

If your app needs large data, mount it under /mnt. Example with a loopback file for local testing:

sudo mkdir -p /mnt/hello-data
# Assuming /dev/loop10 is prepared and formatted
sudo mount /dev/loop10 /mnt/hello-data
sudo chown hello: hello /mnt/hello-data

Document the mount in /etc/fstab only after testing.

8) Clean up safely

sudo systemctl disable --now hello-app
sudo rm -f /etc/systemd/system/hello-app.service
sudo systemctl daemon-reload
sudo userdel hello
sudo rm -rf /etc/hello-app /var/lib/hello-app /var/log/hello-app /var/cache/hello-app /var/spool/hello-app /run/hello-app /usr/local/bin/hello-app /opt/acme/hello-app

Practical examples for common stacks

Example 1: Go or Rust single binary (admin-installed)

• Binary: /usr/local/bin/logship
• Config: /etc/logship/config.yml
• State: /var/lib/logship/
• Logs: /var/log/logship/
• Runtime: /run/logship/
• Temp: /tmp/logship-*
• Service: /etc/systemd/system/logship.service

Install snippet:

sudo install -m 0755 ./logship /usr/local/bin/logship
sudo mkdir -p /etc/logship /var/lib/logship /var/log/logship /run/logship
sudo chown -R logship: logship /var/lib/logship /var/log/logship /run/logship

Example 2: Vendor-bundled Python app under /opt

• Bundle: /opt/vendor/insight-app/{bin, lib,...}
• Symlink for convenience: /usr/local/bin/insight -> /opt/vendor/insight-app/bin/insight
• Config: /etc/insight-app/
• State: /var/lib/insight-app/
• Logs: /var/log/insight-app/

Example 3: Web content served from /srv

• Web root: /srv/www/site
• Logs: /var/log/nginx/site/
• Cache: /var/cache/nginx/
• Config: /etc/nginx/sites-available/site

Tips for troubleshooting and maintenance

• If an app complains about missing write permissions, check whether it is trying to write to /etc or /usr. Redirect writes to /var/lib or /run.
• Keep secrets out of world-readable paths. Use 600 or 640 and a dedicated group.
• Rotate logs in /var/log with your distro's logrotate config.
• Back up /etc and /var/lib. Recreate /var/cache and /run on demand.
• Use find and du to audit space:

  sudo du -sh /var/lib/* | sort -h | tail
  sudo find /var/log -type f -mtime +14 -print

Conclusion

Getting the Linux filesystem hierarchy right makes your services predictable and easy to operate. Start small with a local pilot that places binaries, configs, state, logs, caches, and runtime files in the correct directories. Use dedicated users, strict permissions, and a clear mapping for every artifact. As a next step, package your app for repeatable installs, add backups for /etc and /var/lib, configure log rotation, and monitor disk usage under /var.