Docker Images Production Operations: A Practical Checklist with Examples

Intro

Running Docker images in production is often smoother than the operations around them. The hard part is not writing a Dockerfile; it is knowing what is actually running, where the data lives, how to change a container safely, and how to recover when something goes wrong. This article provides a practical operations checklist for developers, DevOps consultants, and technical startup teams who manage Docker images in production. It connects real commands, expected output, failure signals, and recovery decisions to each operational area.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Each section below covers a specific area: version and environment inventory, safe configuration, verification and diagnostics, failure modes and recovery, and a consolidated operations checklist. The article also includes a dedicated section on common pitfalls, with concrete examples and fixes.

Version and Environment Inventory

Before making any change, you need a complete picture of the Docker environment. Start by collecting the Docker version, storage driver, and the current state of containers and images. This read-only inventory helps you identify what is running, what is stopped, what is consuming resources, and whether the environment matches your expectations.

Run these commands on any Docker host:

docker version --format '{{.Server.Version}}'
docker info --format 'Storage Driver: {{.Driver}}'
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
docker images --format "table {{.Repository}}\t{{.Tag}}\t{{.Size}}"

Example output from docker ps:

NAMES      STATUS          PORTS
web-01     Up 2 hours      0.0.0.0:8080->80/tcp
api-01     Up 2 hours      0.0.0.0:3000->3000/tcp
redis-01   Up 2 hours      6379/tcp

For Docker Compose projects, use docker compose ps, docker compose images, and docker compose config --services to list the defined services. The key is to capture the current state before any change. Note the container names, image tags, ports, and uptime. If a container has been restarting repeatedly, its status will show Restarting (1) 12 seconds ago instead of Up.

Check the installed Docker version because some features are version-specific. For example, BuildKit became the default builder in Docker 23.0. If you are using an older version, some commands or flags may not work. Run docker version on the server and on your local machine if you deploy remotely.

Data Storage Inventory

A critical part of environment inventory is understanding where persistent data lives. Many production incidents happen because data was written to the container's writable layer and lost when the container was removed. To avoid this, list all mounts for each container:

docker inspect --format '{{ .Name }}: {{ range .Mounts }}{{ .Type }}:{{ .Source }} -> {{ .Destination }} {{ end }}' $(docker ps -q)

Example output:

/web-01: volume:/var/lib/docker/volumes/app_data/_data -> /var/lib/app
/api-01: bind:/home/user/data -> /var/lib/app

A named volume such as app_data:/var/lib/app is managed by Docker and survives container recreation. A bind mount such as ./data:/var/lib/app maps a host directory directly. Bind mounts are convenient for development but can cause permission issues in production if the host path is not available on all nodes. As part of the inventory, run a restart test on a non-critical container: stop it, recreate it, and verify the application still sees its files. If the data disappears, the service was writing to the container filesystem instead of a volume or mount.

Safe Configuration Path

Changing a production container configuration can cause downtime or data loss if not done carefully. The safe configuration path means making the smallest justified change to a single scoped item, and always having a way to revert. For example, if you need to update an environment variable, do not rebuild the entire image unless necessary; instead, update the container's configuration and test it.

A typical safe configuration flow for a single container:

1. Record the current configuration: docker inspect <container> --format '{{json .Config.Env}}'.
2. Make a backup of the current container definition if using Compose: cp docker-compose.yml docker-compose.yml.bak.
3. Apply the change. For a simple env var, you can run:

   docker run -d --name new-web-01 -e "LOG_LEVEL=debug" -p 8080:80 myimage:1.4

But in production, it is better to update the Compose file or deployment manifest, then apply it via docker compose up -d.

1. Verify the change: docker inspect new-web-01 --format '{{range .Config.Env}}{{println .}}{{end}}' | grep LOG_LEVEL should show LOG_LEVEL=debug.
2. If the change fails, revert to the backup and recreate the original container.

Avoid passing secrets directly on the command line or in environment variables that can be read via docker inspect. Use Docker secrets (in Swarm) or a secrets manager with Compose. For example, in a Compose file, reference a secret file:

services:
  app:
    image: myapp:latest
    secrets:
      - db_password
secrets:
  db_password:
    file: ./db_password.txt

In Swarm mode, secrets are encrypted and only available to the containers that need them. An owner should be assigned to approve configuration changes. For a startup team, a single DevOps lead or on-call engineer can own configuration changes, with a review every week to check for drift.

Image Tagging and Versioning

A safe configuration path also requires strict image tagging. Never use latest in production because it makes rollbacks impossible. Use a versioning scheme that includes the application version and optionally a build ID. For example, myapp:1.4.2 or myapp:1.4.2-20250315. When you build, tag explicitly:

docker build -t myapp:1.4.2-20250315 .
docker push registry.example.com/myapp:1.4.2-20250315

Then in your Compose file, reference the full tag. If you need to roll back, simply change the tag to the previous version and redeploy.

Verification and Diagnostics

After any change, verify that the system behaves as expected. Diagnostics start with checking container status, logs, and resource usage. Run the following:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
docker logs <container> --tail 100 -f
docker stats --no-stream

The docker stats command shows CPU, memory, and network usage. High memory usage may indicate a leak. The logs often reveal application errors, such as database connection failures.

For deeper inspection, use docker inspect to see environment variables, health checks, and network settings. If the container has a health check defined, retrieve its status:

docker inspect --format '{{json .State.Health}}' <container> | jq .

This returns the health status (healthy, unhealthy, or starting) and recent failure count. If unhealthy, inspect the last output:

docker inspect --format '{{json .State.Health.Log}}' <container> | jq '.[-1].Output'

For Compose services, use docker compose logs -f <service> to follow logs, and docker compose exec <service> sh to run commands inside the container without changing the image. This is useful for checking file permissions or connectivity. For example, to test if the app can reach the database:

docker compose exec app sh -c 'nc -zv db 5432'

If the connection fails, the diagnostic points to a network or database issue, not the app code.

Health Checks

Implement health checks in your Dockerfile or Compose file to automate diagnostics. A health check allows Docker to know when a container is ready to serve traffic and when it is failing. Example for a web app:

services:
  web:
    image: mywebapp:1.0
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

With this health check, Docker will mark the container unhealthy if the endpoint fails three times. In orchestrated environments, unhealthy containers can be automatically restarted or replaced.

Failure Modes and Recovery

Production failures fall into common categories: container exits unexpectedly, application crashes, data loss, or network misconfiguration. Recovery starts with identifying the failure mode and applying the documented recovery procedure.

Container Exits or Restarts

If a container is in a restart loop, check its exit code and logs:

docker ps -a --format "table {{.Names}}\t{{.Status}}\t{{.ExitCode}}"
docker logs <container> --tail 50

An exit code of 137 means the container was killed (often OOM). Increase the memory limit or fix the application's memory usage. An exit code of 1 usually indicates an application error. If the container exits immediately, run it interactively to see the error:

docker run -it --rm myimage:tag sh

Then start the application manually inside the container.

Data Loss

Data loss typically occurs when the container's writable layer was used for persistent data and the container was removed. If you discover that a volume or bind mount was missing, recreate the container with the correct mount. But to recover lost data, you may need to inspect the old container's filesystem if it still exists. For instance, if the container was stopped but not removed, you can copy files out:

docker cp old-container:/var/lib/app/data ./recovered-data

However, if the container is gone, the data is lost unless you have backups. Therefore, always define volumes in your Compose file and back them up regularly. A simple backup command for a named volume:

docker run --rm -v app_data:/data -v $(pwd):/backup alpine tar czf /backup/app_data_backup.tar.gz -C /data .

Network Misconfiguration

Network issues often manifest as containers unable to reach each other. Check the network list and inspect container networks:

docker network ls
docker inspect --format '{{json .NetworkSettings.Networks}}' <container> | jq .

If a container is attached to the wrong network, reconnect it:

docker network connect my-network <container>

For Compose, ensure the service is on the expected network defined in the Compose file. Also check DNS resolution inside the container: docker exec <container> getent hosts <other-service>.

Recovery Runbook

Create a simple runbook for each critical service. A runbook is a document that lists common failures, their symptoms, and step-by-step recovery commands. For example, for a web service, a runbook entry might be:

• Symptom: 502 Bad Gateway from reverse proxy.
• Diagnosis: docker logs web-01 --tail 20 shows "connection refused to upstream".
• Recovery: Check if the app is running: docker ps | grep web. If not, docker compose up -d web. If running but still failing, check the app's health endpoint and database connectivity.

Assign an owner to maintain the runbook for each service. The owner reviews and updates the runbook monthly or after every incident.

Operations Checklist

Use this checklist as a quick reference for daily, weekly, and incident operations.

Daily Checks

• [ ] Run docker ps to see container status; identify any containers in restart loop.
• [ ] Check docker stats --no-stream for abnormal resource usage.
• [ ] Review docker logs <container> --tail 100 for new errors.
• [ ] Verify health check status for all containers.
• [ ] Ensure backup jobs completed successfully (check backup logs).

Weekly Checks

• [ ] Review image sizes: docker images --format "{{.Repository}}:{{.Tag}} {{.Size}}" | sort -k2 -h and clean up unused images with docker image prune (but only if safe).
• [ ] Check for dangling volumes: docker volume ls -f dangling=true.
• [ ] Verify that all running containers use specific tags, not latest.
• [ ] Review configuration drift: compare current environment variables with documentation.
• [ ] Run a restart test on one non-critical container to confirm data persists.

Before Any Change

• [ ] Record current state and relevant logs.
• [ ] Identify the exact component (container, service, image) to change.
• [ ] Determine the blast radius: who is affected if this goes wrong?
• [ ] Prepare a rollback plan: what exact commands revert the change?
• [ ] Assign an owner: the person who will execute and verify the change.
• [ ] Notify stakeholders if the change may cause brief downtime.

After Any Change

• [ ] Verify the expected outcome with a specific command or signal.
• [ ] Check logs for new errors.
• [ ] Confirm health checks are passing.
• [ ] Update the runbook if the procedure changed.
• [ ] Communicate completion to stakeholders.

This checklist should be owned by the operations lead or DevOps engineer. In a startup, one person may own the entire checklist and review it weekly to ensure it is up to date.

Common Pitfalls and How to Avoid Them

Even experienced teams make mistakes with Docker in production. Here are the most common pitfalls, why they happen, and how to avoid or recover from them.

1. Using the latest tag in production

• Why it happens: It is the default and convenient.
• Problem: You cannot know exactly which version is running, and rollback is impossible because latest changes.
• Avoid: Always build and deploy with explicit version tags. Use CI to tag images with a version or commit SHA.
• Recover: If already using latest, immediately tag the current image with a version (docker tag myapp:latest myapp:1.4.2) and update your deployment to use the new tag.

2. Storing data in the container's writable layer

• Why it happens: Developers test locally without volumes and forget to add them in production.
• Problem: Data is lost when the container is removed or updated.
• Avoid: Define volumes or bind mounts for all persistent data paths. Test with a restart before going live.
• Recover: If the container still exists, use docker cp to extract data, then recreate with a volume. If not, restore from backup.

3. Exposing secrets in environment variables or command lines

• Why it happens: It is the quickest way to pass configuration.
• Problem: docker inspect or process listings can reveal secrets. Logs may also capture them.
• Avoid: Use Docker secrets in Swarm, or a secrets manager with Compose. Never pass passwords directly in docker run or Compose environment variables.
• Recover: Rotate the exposed secret, update the application configuration, and ensure no copies remain in shell history or logs.

4. Ignoring health checks

• Why it happens: Health checks are seen as optional or complex to write.
• Problem: Without health checks, Docker cannot tell if a container is actually working, leading to traffic being sent to broken instances.
• Avoid: Add a simple health check to every service. Start with a basic HTTP endpoint check or a command that verifies the process is responsive.
• Recover: Add health checks to your images and redeploy. Or use an external monitoring probe if you cannot modify the image.

5. Not limiting resource usage

• Why it happens: Default Docker settings allow containers to use all host resources.
• Problem: A memory leak in one container can starve others and crash the host.
• Avoid: Set memory and CPU limits in your Compose file or docker run command. For example, mem_limit: 512m in Compose.
• Recover: Immediately set limits and restart the affected container. Monitor memory usage with docker stats.

6. Building images in production

• Why it happens: It seems convenient to build on the production host.
• Problem: The production server accumulates build tools, layers, and potential vulnerabilities. It also makes the environment inconsistent.
• Avoid: Build images in CI/CD and push to a registry. Production only pulls images.
• Recover: Remove build tools from production, pull the built images from the registry, and enforce a policy that no builds happen on production hosts.

7. Not keeping images updated

• Why it happens: Teams are busy and forget to apply security patches.
• Problem: Running old images exposes known vulnerabilities.
• Avoid: Schedule regular image updates, use automated vulnerability scanning, and subscribe to security advisories.
• Recover: Identify outdated images with docker images and check their creation dates, then plan an update window.

8. Overwriting configuration directly on a running container

• Why it happens: Quick fixes such as editing a file inside the container.
• Problem: Changes are lost on container recreation, and the running configuration diverges from the source of truth (Compose file, image).
• Avoid: Never modify running containers directly. Change the Dockerfile or Compose file, then recreate the container.
• Recover: If you made changes directly, document them and apply the same changes to the source files, then recreate the container to verify consistency.

Conclusion

A production-grade Docker images operation is not about memorizing every command; it is about following a consistent, safe process. Start with a complete inventory, make scoped and reversible changes, verify with concrete signals, and have recovery procedures ready. The checklist in this article serves as a starting point for your team's runbook.

Pick one low-risk verification from the Operations Checklist, record the current state, run the documented check, and compare the result with the expected signal. Assign a single owner to each major decision or checklist item, and set a regular review cadence (weekly for checklist health, monthly for runbooks). By doing so, you make failure visible, protect sensitive values, and limit changes to the intended resource. Reliability is built through these repeated, disciplined operations.