Docker Healthchecks Production Operations: A Practical Checklist

Intro

When a containerized service fails in production, the first question is rarely "What broke?" but "How do we know it broke?" Docker healthchecks answer that question by defining a command that the container runs periodically to report its internal state. Yet many teams treat the healthcheck as a checkbox in the Dockerfile or Compose file, and only discover its blind spots during an incident.

This article is a production operations checklist for Docker healthchecks. It is written for developers, DevOps engineers, and technical startup teams who run services in Docker and need a repeatable way to verify that those services are not just running, but actually healthy. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover when the expected state is not reached.

Every section combines a checklist item with at least one concrete command, its expected output, common failure signals, and the recovery decision that follows. The checklist is ordered roughly by how you would approach a production system during normal operations, then during an incident.

Version and Environment Inventory

Before touching anything, know exactly what you are running. A healthcheck is only as good as the runtime that executes it, and a mismatch between what you think is deployed and what is actually deployed is the root cause of many puzzling failures.

Checklist item 1: Record the Docker Engine and Compose plugin versions.

Run:

docker version --format '{{.Server.Version}}'
docker compose version

Expected output example:

24.0.7
Docker Compose version v2.23.3

If the engine is older than 20.10, healthcheck support is missing or behaves differently. Some orchestration features also require recent Compose versions. Record these values in your runbook before issuing any changes.

Checklist item 2: Enumerate running containers and their health status.

Run:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Expected output example:

NAMES       STATUS                    PORTS
api         Up 2 hours (healthy)      0.0.0.0:8080->8080/tcp
worker      Up 2 hours (unhealthy)
redis       Up 2 hours                0.0.0.0:6379->6379/tcp

Notice that worker shows unhealthy while redis shows no health status at all because it has no healthcheck defined. This is your first triage signal: an unhealthy container is a known problem, while a missing healthcheck is a blind spot.

Checklist item 3: Confirm where data is stored before any container replacement.

A container that appears healthy can still lose data if it writes to the container filesystem instead of a mounted volume. Inspect mounts:

docker inspect api --format '{{ range .Mounts }}{{ .Type }} {{ .Name }} {{ .Source }} -> {{ .Destination }}{{ println }}{{ end }}'

Expected output example:

volume app_data /var/lib/docker/volumes/app_data/_data -> /var/lib/app

For Compose projects, run:

docker compose config

This prints the resolved configuration including volumes and healthcheck definitions. If you see a bind mount like ./data:/var/lib/app, remember that it maps a host directory directly. It works on this machine, but if the host path does not exist on another machine, the container will start with an empty directory or fail. Named volumes are more portable and easier to back up.

Checklist item 4: Perform a local restart test before changing production.

On a staging or local environment that mimics production, run the following:

docker compose stop api
docker compose up -d api
docker compose ps
docker exec api ls /var/lib/app

If the file list is empty after the restart, the service was writing to the container filesystem. Production data is already at risk; fix the volume mount before anything else.

Owner and review frequency: The platform or infrastructure owner (for example, "Alex Chen, Infrastructure Lead") is accountable for maintaining the environment inventory. Review the inventory monthly or whenever a new service is deployed.

Safe Configuration Path

Healthcheck configuration lives in the Dockerfile or Compose file. A misconfigured healthcheck can cause the orchestrator to restart a healthy container endlessly, or to mark an unhealthy container as healthy and continue sending it traffic. Change this configuration deliberately.

Checklist item 1: Baseline the current healthcheck configuration.

Read the current definitions before editing:

docker inspect api --format '{{json .Config.Healthcheck}}'

Expected output example:

{"Test":["CMD-SHELL","curl -f http://localhost:8080/health || exit 1"],"Interval":30000000000,"Timeout":5000000000,"Retries":3,"StartPeriod":15000000000}

Note that Docker stores intervals in nanoseconds. A common mistake is to interpret 30000000000 as 30 milliseconds; it is actually 30 seconds. Always verify with docker inspect rather than assuming from the source file.

Checklist item 2: Modify healthcheck only in a controlled file.

If you need to change the healthcheck, edit the Dockerfile or Compose file, not the running container. For example, in docker-compose.yml:

services:
  api:
    image: myapp:2.4
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8080/ready || exit 1"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 40s

Then apply with:

docker compose up -d --no-deps api

The --no-deps flag prevents restarting dependent services unnecessarily, limiting the blast radius.

Checklist item 3: Verify the new healthcheck takes effect.

After applying, check the status over time:

docker ps --format '{{.Names}}: {{.Status}}'

Expected sequence:

api: Up 10 seconds (health: starting)
api: Up 45 seconds (healthy)

If the container flips between starting and unhealthy repeatedly, the start period is too short or the healthcheck command is failing for reasons unrelated to readiness. Inspect the logs:

docker logs api --tail 50

Look for the healthcheck command's own output if it logs anything.

Checklist item 4: Test failure detection deliberately.

Once a container is healthy, simulate a failure to confirm the healthcheck actually catches it. For a web service, you can temporarily break the endpoint by moving a file:

docker exec api mv /app/health /app/health.disabled

Then watch the status change within the configured interval:

docker ps --format '{{.Names}}: {{.Status}}'

Expected output after a few intervals:

api: Up 2 minutes (unhealthy)

This proves the orchestrator will see the failure. Restore the file and confirm recovery. This test is cheap and should be part of any healthcheck change review.

Owner and review frequency: The service owner (for example, "Priya Shah, Engineering Lead") is accountable for the healthcheck configuration of their service. Review any change before merging, and re-evaluate the healthcheck thresholds quarterly or after any significant code change.

Verification and Diagnostics

A healthcheck is a binary signal: healthy or unhealthy. But when it fails, you need to know why. This section is about collecting diagnostics without altering the system.

Checklist item 1: Read the healthcheck command's own execution history.

Docker does not log healthcheck output separately, but you can inspect the container logs for clues. However, a better approach is to run the healthcheck command manually inside the container:

docker exec api curl -f http://localhost:8080/health

If this returns a non-zero exit code, the healthcheck itself is accurate and the service is indeed failing. If it returns 0 but the container is marked unhealthy, the problem is likely a timing issue (interval too short, timeout too low, or start period insufficient).

Checklist item 2: Check process and resource usage inside the container.

Run:

docker stats --no-stream api

Expected output example:

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
f1c1a1a1a1a1   api       78.43%    512.3MiB / 1GiB      50.04%    12.3MB / 45.2MB   0B / 0B           23

High CPU or memory pressure can cause healthcheck commands to time out even if the application is working. Compare with baseline metrics from before the incident.

Checklist item 3: Validate the healthcheck command's dependencies.

Many healthchecks rely on tools like curl, wget, or pg_isready. If the image does not include them, the healthcheck always fails. Run:

docker exec api which curl

If the command is not found, you have two options: either replace the healthcheck with a command that uses built-in tools (for example, node -e "require('http').get('http://localhost:8080/health', r => process.exit(r.statusCode === 200 ? 0 : 1)).on('error', () => process.exit(1))" for Node.js), or add the missing tool to the image. The first option is often better for production because it avoids increasing the image size for a single check.

Checklist item 4: For Compose services, review the entire stack's health status.

Run:

docker compose ps --format "table {{.Name}}\t{{.Status}}\t{{.Service}}"

Expected output example:

Name             Status                    Service
myapp_api_1      Up 2 hours (healthy)      api
myapp_worker_1   Up 2 hours (unhealthy)    worker
myapp_redis_1    Up 2 hours                redis

An unhealthy worker may not directly affect the API, but it may signal a problem with a shared dependency like the database. Check logs for the worker:

docker compose logs --tail 200 worker

Look for stack traces, connection errors, or missing environment variables.

Owner and review frequency: The on-call engineer or reliability team is accountable for running diagnostics during an incident. After resolution, the post-incident review should verify that the healthcheck thresholds and command are still appropriate and that no new blind spots were introduced.

Failure Modes and Recovery

Healthchecks themselves rarely fail; it is the service behind them that fails. But an incorrectly designed healthcheck can create a failure mode of its own: a healthcheck that is too strict causes unnecessary restarts, while one that is too lenient allows broken services to keep receiving traffic. Here are the common failure modes and how to recover.

Failure mode 1: Healthcheck flapping and restart loops.

Why it happens: The healthcheck interval is shorter than the application's startup time, or the timeout is too low for the check to complete under normal load, or the healthcheck command intermittently fails due to transient dependencies.

How to detect: Container status alternates between unhealthy and starting repeatedly. Logs show frequent restarts.

How to recover:

1. Determine the actual startup time by observing the container from a clean start.
2. Increase start_period to at least 1.5 times the observed startup time.
3. If the check still fails during normal operation, increase timeout or decrease interval.
4. Example adjustment:

healthcheck:
  test: ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]
  interval: 10s
  timeout: 5s
  retries: 5
  start_period: 90s

Failure mode 2: Orphaned containers marked healthy after process death.

Why it happens: If the main process dies but the container does not exit (for example, a zombie process or a PID 1 that ignores signals), the healthcheck may still pass if it checks a static endpoint. The service is effectively down, but Docker reports healthy.

How to detect: External monitoring shows failures while Docker reports healthy. Inspect the main process:

docker inspect api --format '{{.State.Status}} {{.State.Pid}}'

If the PID is 1 or a zombie, the main process is not running.

How to recover: Redesign the healthcheck to verify actual application logic, not just a static file. For example, make the health endpoint perform a trivial database query or check a critical in-memory state. Alternatively, use a process supervisor as PID 1 (like tini or dumb-init) so that when the main process dies, the container exits and the orchestrator restarts it.

Failure mode 3: Healthcheck passes but service is degraded.

Why it happens: The healthcheck only tests a simple endpoint, but a downstream dependency (database, cache, external API) is down. The service can respond to health checks but cannot serve real requests.

How to detect: Error rates increase in application logs, but health status remains healthy.

How to recover: Implement a deeper healthcheck that includes a lightweight test of each critical dependency. For example:

curl -f http://localhost:8080/health/deep

The endpoint could return 200 only if all dependencies are reachable within a short timeout. Alternatively, use two healthchecks: a shallow one for liveness (used by the orchestrator) and a deeper one for readiness (used by a load balancer) if your orchestration supports it.

Failure mode 4: Healthcheck command missing from image.

Why it happens: The image was built on a minimal base that does not include curl, wget, or other tools assumed by the healthcheck. The healthcheck always fails with exit code 127 (command not found).

How to detect: The container is immediately marked unhealthy after start, and inspecting logs shows something like OCI runtime exec failed: exec failed: unable to start container process: exec: "curl": executable file not found in $PATH.

How to recover: Replace the healthcheck with a command that uses built-in language or OS tools. For Python:

healthcheck:
  test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/health').read()"]

For Node.js:

healthcheck:
  test: ["CMD", "node", "-e", "require('http').get('http://localhost:8080/health', r=>process.exit(r.statusCode===200?0:1)).on('error', ()=>process.exit(1))"]

These avoid adding extra packages to the image.

Owner and review frequency: The service owner is responsible for fixing the healthcheck based on the failure mode. After recovery, update the runbook with the new healthcheck configuration and the lessons learned. Review healthcheck effectiveness quarterly.

Operations Checklist

This is a condensed daily/weekly operations checklist for teams running Docker healthchecks in production. It assumes you have completed the configuration and verification steps above.

Daily checks (on-call engineer):

• [ ] Run docker ps --format "table {{.Names}}\t{{.Status}}" and confirm no unexpected unhealthy or starting states.
• [ ] For each service with a healthcheck, confirm the status has been stable for the last hour (use monitoring dashboards if available).
• [ ] Check container logs for healthcheck-related errors: docker logs <container> --since 1h | grep -i health
• [ ] If any container is unhealthy, follow the diagnostics in the Verification and Diagnostics section and open an incident if needed.

Weekly checks (service owner):

• [ ] Review healthcheck configurations against current service behavior. Pay attention to new endpoints or changed timeouts.
• [ ] Run a failure injection test on one service: intentionally break the healthcheck and verify the orchestrator marks the container unhealthy, then restore.
• [ ] Confirm that all critical services have a healthcheck defined. Identify any container with no health status and add one if appropriate.
• [ ] Update the runbook with any changes to healthcheck commands, intervals, or thresholds.

Monthly checks (infrastructure owner):

• [ ] Verify Docker Engine and Compose plugin versions are still supported and compatible with healthcheck features.
• [ ] Audit volume and bind mount usage for all containers to prevent hidden data loss.
• [ ] Review the healthcheck failure modes from the past month and identify patterns.
• [ ] Ensure that healthcheck commands do not depend on tools that are not guaranteed in the base image.

Owner and review frequency: Each checklist item has a named owner as indicated. The daily checks are owned by the on-call engineer, the weekly checks by the service owner, and the monthly checks by the infrastructure owner. The checklist itself is reviewed and updated after every major incident or at least quarterly.

Common Pitfalls

Even with a solid checklist, teams repeatedly fall into the same traps. Here are the most common ones and how to avoid them.

Pitfall 1: Using healthchecks for readiness instead of liveness.

Docker's healthcheck is a liveness and basic readiness mechanism, but it is not a full readiness probe. If you use it to gate traffic to a service that takes a long time to warm up, you may either route traffic too early or restart too often. Use start_period to cover initial warm-up, and rely on a load balancer or service mesh for more nuanced readiness checks.

Pitfall 2: Hardcoding secrets in healthcheck commands.

A healthcheck in a Compose file or Dockerfile may need credentials to call an endpoint. Avoid embedding secrets directly; use environment variables or Docker secrets, and ensure the value is not printed in logs. Example:

healthcheck:
  test: ["CMD-SHELL", "curl -f -H 'Authorization: Bearer $HEALTH_TOKEN' http://localhost:8080/health || exit 1"]

Set HEALTH_TOKEN via environment or secret store.

Pitfall 3: Ignoring healthcheck for stateless helpers.

Some teams skip healthchecks for stateless workers or sidecars because they "can't be unhealthy." But a worker that cannot connect to the database is unhealthy in practice. Define a simple check, for example:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -h db -U user -d app || exit 1"]
  interval: 30s
  timeout: 5s
  retries: 3

Pitfall 4: Not testing the healthcheck in CI.

The healthcheck is code, and it can break. Include a step in your CI pipeline that builds the image, starts the container, and asserts that it becomes healthy. A simple smoke test:

docker run -d --name test_health myapp:latest
for i in $(seq 1 30); do
  status=$(docker inspect --format '{{.State.Health.Status}}' test_health)
  if [ "$status" = "healthy" ]; then exit 0; fi
  sleep 2
done
exit 1

Pitfall 5: Setting too aggressive intervals.

An interval of 1 second with a timeout of 1 second may cause more harm than good. The healthcheck itself consumes CPU and can add load to the service. Choose an interval that is meaningful for your service: for a typical web app, 10 to 30 seconds is reasonable. Less frequently for batch jobs.

Conclusion

Docker healthchecks are a small part of your container runtime, but they are the first line of defense for service availability. A well-designed healthcheck tells you the truth about your service without overloading it, and it gives the orchestrator the signal it needs to keep traffic flowing to healthy instances. A poorly designed healthcheck either lies to you or causes churn.

This article has walked through a practical operations checklist: start with an environment inventory to know what you have, follow a safe configuration path for changes, diagnose problems with concrete commands, understand common failure modes and how to recover, and embed the checks into daily, weekly, and monthly operations. The common pitfalls section should help you avoid the mistakes that others have already made.

The next step is to pick one low-risk service and run the version and environment inventory commands from this article. Record the current state, check the healthcheck definition, and deliberately inject a failure to see how your system responds. Then decide if your healthcheck thresholds and commands still match the reality of your service. If they do not, adjust them using the safe configuration path described here. Remember: a healthcheck is only useful if you trust its signal in an emergency. Build that trust now, before you need it.