## Intro Docker healthchecks are the first line of defense for keeping containers reliable in production. A well-defined healthcheck tells Docker whether a container is actually functional, not just running. Without it, a container can sit in an 'up' state while the application inside is dead to incoming traffic. This article shows you how to define healthchecks, monitor their output, alert on unhealthy states, and recover from common failures—all with practical commands and examples you can run today. This guide is written for developers, DevOps engineers, and startup teams who run Docker in production or staging. You will learn to move from 'the container is up' to 'the service is verified healthy' using built-in Docker features and a small amount of scripting. Every example is tested against Docker Engine 24.0 and Docker Compose v2, which are current as of early 2025. We will cover version and environment checks, safe configuration, verification and diagnostics, failure modes, and an operations checklist. Along the way, you will see real commands, sample output, and concrete recovery steps. The goal is operational safety: observe before changing, limit blast radius, avoid hardcoding secrets, and verify every fix. ## Version and Environment Inventory Before adding or changing a healthcheck, confirm what Docker version and storage driver you are using. Run: ```bash docker version --format '{{.Server.Version}}' ``` Expected output: `24.0.7` or similar. If you use Docker Compose, check: ```bash docker compose version ``` Expected output: `Docker Compose version v2.24.0`. Both commands are read-only and safe to run on any host. Next, list running containers and their current health status if defined: ```bash docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" ``` Sample output: ``` NAMES STATUS PORTS web Up 10 minutes (healthy) 0.0.0.0:8080->80/tcp db Up 10 minutes (unhealthy) 5432/tcp ``` Notice the `(healthy)` and `(unhealthy)` suffixes. These appear only if the image defines a healthcheck or you provided one at runtime. If a container lacks a healthcheck, Docker cannot know its internal state. To see the full health configuration for a container, use `docker inspect`: ```bash docker inspect --format '{{json .Config.Healthcheck}}' web ``` Sample output: ```json {"Test":["CMD-SHELL","curl -f http://localhost/ || exit 1"],"Interval":30000000000,"Timeout":5000000000,"Retries":3} ``` This shows the healthcheck command, interval (30 seconds), timeout (5 seconds), and retry count (3). Note that Docker stores durations in nanoseconds, but `docker inspect` with `--format` makes them readable. If you run Compose, get service-level health with: ```bash docker compose ps ``` Sample output: ``` NAME COMMAND SERVICE STATUS PORTS project-web-1 "nginx -g 'daemon of…" web running (healthy) 0.0.0.0:8080->80/tcp project-db-1 "docker-entrypoint.s…" db running (healthy) 5432/tcp ``` ### Data Persistence and Healthchecks A healthcheck often reads or writes a file to prove the application is alive. Where that file lives matters. If the file is written to the container's writable layer, it disappears when the container is recreated. Always use a named volume or bind mount for persistent state. Example Compose snippet with a named volume: ```yaml services: app: image: myapp:1.0 volumes: - app_data:/var/lib/app volumes: app_data: ``` To test persistence locally, stop the container and recreate it: ```bash docker stop app_container docker rm app_container docker compose up -d app ``` Then check the data is still present: ```bash docker compose exec app ls /var/lib/app ``` If the directory is empty, the original container was probably writing to the writable layer instead of the volume. ## Safe Configuration Path Adding a healthcheck should be a controlled change. Start by writing the healthcheck in a Compose file or Dockerfile, not directly on the command line, so it is version-controlled and reviewable. ### Healthcheck Options A healthcheck can be defined in a Dockerfile: ```dockerfile FROM nginx:latest HEALTHCHECK --interval=30s --timeout=5s --retries=3 \ CMD curl -f http://localhost/ || exit 1 ``` Or in a Compose file: ```yaml services: web: image: nginx:latest healthcheck: test: ["CMD", "curl", "-f", "http://localhost/"] interval: 30s timeout: 5s retries: 3 start_period: 10s ``` Key parameters: - `test`: the command to run. Use `CMD` for exec form or `CMD-SHELL` for shell form. - `interval`: how often to run the check (default 30s). - `timeout`: maximum time for the check to complete (default 30s). - `retries`: consecutive failures needed to mark unhealthy (default 3). - `start_period`: grace period before checks count (default 0s). Useful for slow-starting apps. A good healthcheck command is lightweight and fails fast. Avoid heavy operations like database migrations or external API calls. For a web service, `curl -f http://localhost/health` is common. For a database, use a client ping: ```yaml db: image: postgres:16 healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres"] interval: 10s timeout: 5s retries: 5 ``` For a Redis container: ```yaml redis: image: redis:7 healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 10s timeout: 3s retries: 3 ``` ### Protecting Secrets in Healthchecks Never hardcode credentials in healthcheck commands. If your healthcheck needs to authenticate, use environment variables or Docker secrets. For example, a database healthcheck with a password: ```yaml db: image: postgres:16 environment: POSTGRES_PASSWORD_FILE: /run/secrets/db_password secrets: - db_password healthcheck: test: ["CMD-SHELL", "pg_isready -U postgres && PGPASSWORD=$(cat /run/secrets/db_password) psql -U postgres -c 'SELECT 1' >/dev/null 2>&1"] secrets: db_password: file: ./db_password.txt ``` The password is read from a secret file at runtime, never committed to source control. ### Incremental Rollout When adding a healthcheck to an existing service, consider enabling it in staging first, observe for a few days, then promote to production. A healthcheck that is too strict can cause unnecessary restarts or mark healthy containers as unhealthy. A common mistake is setting `interval` too low. Every healthcheck execution consumes CPU and may add load to the application. For most web services, 10-30 seconds is reasonable. Adjust based on the app's startup time and traffic patterns. ## Verification and Diagnostics Once a healthcheck is in place, verify it works as expected. Start by checking the health status directly: ```bash docker inspect --format '{{.State.Health.Status}}' web ``` Expected output: `healthy`. To see the history of healthcheck results: ```bash docker inspect --format '{{json .State.Health}}' web | jq ``` Sample output: ```json { "Status": "healthy", "FailingStreak": 0, "Log": [ { "Start": "2025-03-25T10:00:00.123456789Z", "End": "2025-03-25T10:00:00.234567890Z", "ExitCode": 0, "Output": " % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0\r100 6154 100 6154 0 0 135k 0 --:--:-- --:--:-- --:--:-- 135k\n" } ] } ``` ExitCode 0 means success. Non-zero means failure. The Output field can help debug why a healthcheck is failing. If a container is unhealthy, get its logs: ```bash docker logs --tail 50 unhealthy_container ``` Look for application errors, missing dependencies, or port conflicts. ### Debugging Inside the Container Sometimes you need to run the healthcheck command manually inside the container to see its output: ```bash docker exec -it web sh # inside container curl -f http://localhost/ || echo "healthcheck failed" ``` If curl is not installed, you may need to use wget or the application's CLI. Some images are minimal and do not include curl; in that case, consider switching to a different base image or using a healthcheck that relies on built-in tools. For Compose services, use: ```bash docker compose exec web sh ``` For a database container, test connectivity: ```bash docker exec db pg_isready -U postgres ``` Expected output: `postgres:5432 - accepting connections`. ### Healthcheck and Restart Policies Healthchecks interact with restart policies. If a container is `unhealthy` and has a restart policy like `unless-stopped`, Docker does not automatically restart it just because it is unhealthy. Restarts happen only when the container exits. However, some orchestrators like Docker Swarm can replace unhealthy replicas. In standalone Docker, you need external tooling to act on unhealthy status. A simple way to restart unhealthy containers is a cron job or a monitoring script. For example: ```bash #!/bin/bash # restart unhealthy containers every minute for c in $(docker ps -q --filter "health=unhealthy"); do echo "Restarting $c" docker restart $c done ``` Caution: blindly restarting can mask underlying issues. Use this as a stopgap while you fix the root cause. ## Failure Modes and Recovery Healthchecks can fail for many reasons. Understanding common failure modes helps you respond quickly. ### 1. Healthcheck command not found If the image lacks the tool used in the healthcheck, every check fails. For example, using `curl` in an Alpine-based image that does not have curl installed. Why it happens: The healthcheck command is not present in the container's filesystem. How to avoid: Use commands available in the base image, or install the required tool in the Dockerfile. For Alpine, use `wget` or `busybox wget` as an alternative. Recovery: Update the healthcheck to use an available command or install the tool. ### 2. Healthcheck times out If the check takes longer than `timeout`, it is considered failed. Why it happens: The command is too slow, the application is overloaded, or the network is slow. How to avoid: Set a reasonable timeout (5-10 seconds). Optimize the healthcheck command to be fast. Recovery: Increase timeout temporarily, but investigate why the check is slow. ### 3. Healthcheck returns wrong exit code Your healthcheck script may exit 0 even when the service is down, or vice versa. Why it happens: The script logic is flawed, e.g., not properly checking the response body or only checking if the port is open. How to avoid: Test the healthcheck command manually during development. Ensure it returns non-zero on failure. Recovery: Fix the script to accurately reflect service health. ### 4. Container marked unhealthy due to start_period not set Fast-failing checks during startup can mark the container unhealthy before the app is ready. Why it happens: The healthcheck runs before the app finishes initializing. How to avoid: Set `start_period` to cover the expected startup time. Recovery: Add or increase `start_period`. ### 5. Healthcheck causes high CPU load If the interval is too short and the command is heavy, it can consume significant resources. Why it happens: Healthcheck running too frequently or doing expensive operations. How to avoid: Use lightweight checks, set interval to at least 10s, and avoid complex shell commands. Recovery: Simplify the healthcheck and increase interval. ### 6. Healthcheck interferes with application behavior Some healthchecks perform write operations (e.g., creating a temp file) that may affect the app. Why it happens: The healthcheck command has side effects. How to avoid: Use read-only checks whenever possible. Recovery: Change to a read-only check. ## Monitoring and Alerting Integration Docker healthchecks alone do not provide alerting. You need to integrate them with a monitoring system. Below are examples using simple scripts and common tools. ### Basic Monitoring with Docker Events Docker emits events when a container's health status changes. You can listen for these events: ```bash docker events --filter event=health_status ``` Sample output: ``` 2025-03-25T10:00:00.123456789Z container health_status: unhealthy 1234567890abcdef 2025-03-25T10:00:30.123456789Z container health_status: healthy 1234567890abcdef ``` You can pipe this into a script that sends alerts via webhook or email. ### Using a Cron Job to Check Health and Alert A simple cron job can check the health of all containers and send a notification if any are unhealthy: ```bash #!/bin/bash unhealthy=$(docker ps -q --filter "health=unhealthy") if [ -n "$unhealthy" ]; then echo "Unhealthy containers: $unhealthy" | mail -s "Docker health alert" ops@example.com fi ``` Schedule it every minute with cron. ### Integration with Prometheus and Alertmanager For more robust monitoring, use cAdvisor or Docker's Prometheus metrics endpoint. Docker daemon can expose metrics on `127.0.0.1:9323/metrics`, including health status. Configure Prometheus to scrape and Alertmanager to alert. Example Prometheus alert rule: ```yaml groups: - name: docker-health rules: - alert: ContainerUnhealthy expr: docker_container_health_status == 0 for: 5m labels: severity: critical annotations: summary: "Container {{ $labels.name }} is unhealthy" ``` This requires the Docker daemon metrics to be enabled and a Prometheus exporter that reports health status. ### Dashboarding with Grafana Grafana can visualize container health status over time using Prometheus data. Create a panel showing count of unhealthy containers per service. A simple dashboard query: ```promql count(docker_container_health_status == 0) by (name) ``` This gives a real-time view of unhealthy containers. ## Operations Checklist Use this checklist before and after deploying containers with healthchecks. Assign an owner for each item (e.g., DevOps lead) and schedule a weekly review of the healthcheck configuration. | Item | Owner | Frequency | Command / Check | |------|-------|-----------|-----------------| | Confirm Docker version | DevOps Lead | Monthly | `docker version` | | Verify healthcheck exists for each critical container | Service Owner | Weekly | `docker inspect --format '{{.Config.Healthcheck}}' ` | | Check current health status of all containers | Ops Engineer | Daily | `docker ps --filter "health=unhealthy"` | | Review healthcheck logs for repeated failures | Ops Engineer | Weekly | `docker inspect --format '{{json .State.Health}}' ` | | Test healthcheck command manually in staging | Developer | Before release | `docker exec ` | | Validate secret handling (no plaintext secrets in healthcheck) | Security Lead | Quarterly | `docker inspect --format '{{.Config.Healthcheck.Test}}' ` | | Update alert rules as services change | Monitoring Admin | Monthly | Review alert definitions | Each item should have a documented recovery step if it fails. For example, if healthchecks are missing, add them to the Dockerfile or Compose file and redeploy. ## Common Pitfalls and Mistakes Beyond the failure modes, here are pitfalls teams often encounter: 1. **Not setting a healthcheck at all.** Many containers run without healthchecks. Without them, Docker cannot tell if the app is functional. 2. **Using the same healthcheck for different services.** A web server healthcheck does not work for a database. Customize per service. 3. **Healthcheck that depends on network access to external services.** If the healthcheck requires internet access, it may fail when only internal network is available. 4. **Ignoring unhealthy containers.** Some teams see `unhealthy` but do nothing because the container is still running. This defeats the purpose. 5. **Too many retries or long intervals.** This delays detection of failure. 6. **Not testing healthchecks in CI/CD.** Always run a container in CI and verify it becomes healthy. 7. **Forgetting to update healthchecks when the app changes.** If the health endpoint changes path, the healthcheck will fail. How to avoid: include healthcheck verification in your CI pipeline. For each service, run the container, wait for it to become healthy, and fail the build if it does not. ## Conclusion Docker healthchecks are a simple yet powerful tool for ensuring containerized applications are truly available. By defining clear healthchecks, monitoring them, and setting up alerts, you can catch issues before users do. The commands and examples in this article give you a solid foundation. Start by adding a healthcheck to one critical service, observe its behavior, and gradually roll out to the rest. Remember: a healthcheck is only useful if you act on its signals. Integrate it with your monitoring stack and incident response process. For next steps, pick a container that currently lacks a healthcheck, add one using the guidance above, and verify it reports healthy. Then set up a simple alert using a cron job or Prometheus. Finally, document the recovery steps for when the healthcheck fails. With these practices, you will move from reactive troubleshooting to proactive reliability.