Docker Healthchecks: A Local Lab Setup with Practical Examples

Intro

Docker healthchecks are a built-in way to tell whether a container is actually healthy, not just running. A container can be up while its application is hung, crash-looping internally, or unable to reach a dependency. Healthchecks let Docker and orchestrators detect that state and act on it, for example by restarting the container or routing traffic away from it.

This article is a hands-on guide to building a local lab where you can experiment with Docker healthchecks safely. It covers version and environment inventory, safe configuration, verification, failure modes and recovery, and an operations checklist. Each section includes concrete commands, expected output, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, protect secrets, verify results, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before configuring healthchecks, know what you are working with. Run:

docker version

Expected output includes client and server versions, for example Client: Docker Engine - Community 24.0.5 and Server: Docker Engine - Community 24.0.5. Healthcheck support has existed since Docker 1.12, but newer versions may include fixes or additional options. The Docker Compose healthcheck syntax is consistent across v2 and v3 of the Compose file format, but if you are using an older Compose version, check compatibility.

To see what is currently running and whether healthchecks are already in use:

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

Look for statuses like Up 5 minutes (healthy) or Up 5 minutes (unhealthy). If you see (healthy) or (unhealthy), a healthcheck is already defined. If you only see Up 5 minutes, there is no healthcheck.

For a deeper look at a container's configuration, mount points, networks, and health status, use:

docker inspect <container> | grep -A 10 Health

This shows the healthcheck definition and last results. For Compose projects, use:

docker compose ps

to see service health, and:

docker compose logs -f <service>

to follow logs and catch healthcheck failures in real time.

Data Persistence Check

One production-like test you should run locally is a restart test. If your application stores data, confirm where files are written before changing anything. A named volume like app_data:/var/lib/app is managed by Docker and easier to reuse across container rebuilds. A bind mount like ./data:/var/lib/app maps a host directory directly and is convenient for development, but can cause permission, portability, and backup issues if the same path does not exist on another machine.

To test persistence, stop the container, recreate it, and check if the data is still present:

docker stop my_app
docker run -d --name my_app -v app_data:/var/lib/app my_image
docker exec my_app ls /var/lib/app

If the directory is empty, the service was likely writing to the container filesystem instead of the volume.

Safe Configuration Path

After understanding your environment, add a healthcheck to a container. The safest way is to modify the Dockerfile or Compose file rather than overriding at runtime, so the healthcheck is versioned and reproducible.

Dockerfile Healthcheck

Here is a simple Dockerfile healthcheck for a web service using curl:

FROM nginx:alpine
RUN apk add --no-cache curl
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
  CMD curl -f http://localhost/ || exit 1

Explanation:

• interval=30s: run the check every 30 seconds.
• timeout=3s: each check must complete within 3 seconds.
• start-period=10s: give the container 10 seconds to start before considering failures.
• retries=3: after 3 consecutive failures, mark the container unhealthy.
• CMD curl -f http://localhost/ || exit 1: the healthcheck command; curl -f fails on HTTP errors.

Docker Compose Healthcheck

If you use Compose, healthchecks can be defined per service:

services:
  web:
    image: nginx:alpine
    ports:
      - "80:80"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 10s

Make sure the healthcheck command is available inside the container. Alpine images often need curl added, while Debian-based images may include wget. Test the command manually before relying on it:

docker exec my_web curl -f http://localhost/

If the command fails or is missing, adjust the image or healthcheck.

Healthcheck Command Options

Beyond curl, common healthcheck commands include:

• For a database, use pg_isready -U postgres in the postgres image.
• For a Redis instance, use redis-cli ping and expect PONG.
• For a custom application, write a small script that checks an HTTP endpoint or performs a small SQL query.

Always make the healthcheck deterministic and side-effect free. It should not modify data.

Verification and Diagnostics

Once a healthcheck is configured, verify it works. Run:

docker ps

Within a few seconds to a minute, you should see the container status change to (healthy). If it shows (unhealthy), investigate immediately.

Check recent healthcheck results:

docker inspect --format='{{json .State.Health}}' my_container | jq

This outputs a JSON block including the last few exit codes and output. For example:

{
  "Status": "unhealthy",
  "FailingStreak": 3,
  "Log": [
    {
      "Start": "2023-05-01T12:00:00.000000000Z",
      "End": "2023-05-01T12:00:00.500000000Z",
      "ExitCode": 1,
      "Output": "curl: (7) Failed to connect to localhost port 80: Connection refused\n"
    }
  ]
}

A non-zero exit code indicates the healthcheck command failed. The Output field often tells you why.

Common failure reasons:

• Wrong port or path.
• Missing dependencies (curl not installed).
• Application not ready within start period.
• Healthcheck command permissions.

To debug, run the exact healthcheck command inside the container:

docker exec my_container sh -c "curl -f http://localhost/"

Observe the exit code and output. Adjust one variable at a time, such as the interval or timeout, and re-check.

Failure Modes and Recovery

Healthchecks can themselves fail in ways that affect application availability. Understand these failure modes and how to recover.

False Positives (Healthy When Broken)

A healthcheck may pass even when the application is not fully functional. For example, a curl to / returns 200 but the backend is down. To avoid this, have the healthcheck exercise a critical path, such as a /healthz endpoint that checks database connectivity.

Recovery: refine the healthcheck to test deeper functionality.

False Negatives (Unhealthy When Working)

If the healthcheck is too strict, Docker may restart a healthy container unnecessarily. For example, a timeout of 1 second on a slow endpoint. This causes container restarts and potential downtime.

Recovery: increase timeout or interval, or make the healthcheck command more efficient.

Healthcheck Command Side Effects

A healthcheck should be read-only. If it writes data, it may cause contention or inconsistent state. For instance, a healthcheck that runs a large query or updates a table.

Recovery: change the healthcheck to a simple, read-only operation.

Healthcheck and Restart Policies Interaction

Docker restart policies (like --restart=unless-stopped) do not restart a container that is unhealthy; they only restart containers that have exited. However, orchestration tools like Docker Swarm or Kubernetes use healthchecks to reschedule unhealthy services. Do not assume a restart policy will handle unhealthy containers.

Recovery: combine healthchecks with a process manager or orchestrator that acts on health status.

Common Pitfalls

1. Forgetting the start period: Some applications take time to start. Without --start-period, the healthcheck may mark the container unhealthy before it is ready, causing unnecessary restarts. Fix: set a realistic start period based on your app's startup time.

1. Using a healthcheck command that is not in the image: Many minimal images lack curl or wget. Fix: either install the tool in the Dockerfile or use a different healthcheck method, such as a script or a built-in binary.

1. Not testing the healthcheck in a local environment: Deploying healthchecks directly to production without local testing can lead to unexpected behavior. Fix: always test in a local lab, observe the status transitions, and simulate failures.

1. Ignoring healthcheck logs: The health status alone does not tell you why it is failing. Fix: inspect the health logs with docker inspect and include the output in your monitoring.

1. Healthcheck doing too much: A healthcheck that is heavy (e.g., running a large query) can slow down the container. Fix: keep it lightweight and fast.

Operations Checklist

Use this checklist before and after deploying healthchecks.

Conclusion

Docker healthchecks are a powerful tool for improving container reliability, but they need to be configured carefully and tested locally. This article has provided a structured approach: start with an environment inventory, configure healthchecks safely, verify them, understand failure modes, and follow an operational checklist.

As a next step, take one container in your local environment and add a healthcheck using the examples above. Observe the status in docker ps, inspect the health logs, and then simulate a failure by making the healthcheck command fail. Document the expected behavior and recovery steps.

A reliable healthcheck setup makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.