Docker Healthchecks: Common Errors and Practical Fixes with Real-World Examples

Intro

Docker healthchecks are a critical tool for ensuring that your containers are not just running, but actually functioning correctly. Without them, Docker can only tell you if the container process is alive; it cannot tell you if your application is stuck in an infinite loop, lost its database connection, or is otherwise unhealthy. Healthchecks let you define a command that Docker runs inside the container at regular intervals to determine if the container is healthy. When a healthcheck fails, Docker marks the container as unhealthy and can even restart it automatically (depending on restart policies).

However, healthchecks are often misconfigured, leading to confusing error messages, containers flapping between healthy and unhealthy states, or healthchecks that never report success. These errors can cause containers to be restarted unnecessarily, or worse, to be considered healthy when they are actually broken. This article will help you diagnose and fix common Docker healthcheck errors with practical examples. We will cover how healthchecks work, typical misconfigurations, safe configuration patterns, verification steps, failure modes and recovery, and an operations checklist to keep your containers healthy in production.

By the end of this article, you will be able to:

• Understand the healthcheck mechanism and how it reports status.
• Identify common error messages and their causes.
• Configure healthchecks safely in Dockerfile and Docker Compose.
• Verify healthchecks locally and interpret results.
• Recover from failed healthchecks and avoid common pitfalls.

Let us dive in.

Version and Environment Inventory

Before we start fixing healthchecks, let us establish a clear baseline. Healthcheck behavior can vary slightly between Docker versions, so it helps to know what you are working with. This section covers the essential version and environment details you need to collect.

Prerequisites:

• Docker Engine 1.12 or later (healthcheck feature was introduced in 1.12).
• Basic familiarity with Docker commands and Dockerfile syntax.
• A container image to test with (we will use nginx and postgres as examples).

Check your Docker version: Run the following command on your Docker host:

docker version

Expected output snippet:

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 ...
Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
 ...

Make sure your server version is at least 1.12 to support healthchecks.

Environment Topology: For this article, we assume a single Docker host running Linux containers (the examples are applicable to Windows containers with minor path adjustments). We will use the Docker CLI and Docker Compose v2. No external orchestration like Kubernetes is needed; healthcheck concepts are native to Docker.

Example Container Images: We will use the official nginx image because it includes curl in recent versions (required for HTTP-based healthchecks), and postgres because it includes pg_isready. If your image lacks a suitable healthcheck command, you may need to install tools like curl or wget in your image (covered later).

Now that we have our environment set, let us understand how healthchecks are configured.

Safe Configuration Path

Configuring a healthcheck seems simple: provide a command that returns 0 for healthy and non-zero for unhealthy. However, many errors arise from improper configuration. In this section, we will cover the syntax, common mistakes, and safe patterns for writing healthchecks in Dockerfiles and Docker Compose files.

Healthcheck Syntax in Dockerfile

The HEALTHCHECK instruction in a Dockerfile has two forms:

• HEALTHCHECK [OPTIONS] CMD command - runs a command inside the container.
• HEALTHCHECK NONE - disables any healthcheck inherited from the base image.

Options include:

• --interval=DURATION (default: 30s)
• --timeout=DURATION (default: 30s)
• --start-period=DURATION (default: 0s)
• --retries=N (default: 3)

Example Dockerfile snippet:

FROM nginx:alpine
HEALTHCHECK --interval=5m --timeout=3s \
  CMD curl -f http://localhost/ || exit 1

Common error: command not found. If the healthcheck command relies on a tool that is not installed in the image, you will see an error in docker inspect output like:

"Health": {
  "Status": "unhealthy",
  "FailingStreak": 1,
  "Log": [
    {
      "Start": "2023-09-01T12:00:00.000Z",
      "End": "2023-09-01T12:00:00.500Z",
      "ExitCode": 127,
      "Output": "/bin/sh: curl: not found\n"
    }
  ]
}

Exit code 127 means command not found. To fix, either install the tool (RUN apk add --no-cache curl for Alpine) or use a built-in alternative. For example, if your image has wget, use wget -q --spider http://localhost/.

Safe pattern for HTTP healthcheck:

FROM nginx:alpine
RUN apk add --no-cache curl
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD curl -f http://localhost/ || exit 1

Note: -f makes curl fail on HTTP errors (4xx/5xx) with non-zero exit code.

Healthcheck Syntax in Docker Compose

In Docker Compose, you define healthcheck under the service:

services:
  web:
    image: nginx:alpine
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 10s

The test can be a string or a list. If a string, it is run with /bin/sh -c; if a list, it is executed directly without a shell (recommended for clarity and avoiding shell interpretation issues).

Common error: misinterpreting exit codes in list form. When using a list, the command is executed without a shell. So ["CMD", "curl -f http://localhost/"] would fail because it tries to execute a binary named curl -f http://localhost/ as a single argument. Use separate arguments: ["CMD", "curl", "-f", "http://localhost/"].

Safe pattern for Compose healthcheck:

services:
  web:
    image: nginx:alpine
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s

start_period gives the container time to start before healthchecks begin failing and counting toward retries.

Healthcheck for Dockerfile vs Compose

If both are defined, Compose overrides the Dockerfile healthcheck. It is recommended to define the healthcheck in the Dockerfile for reusable images, and override in Compose only when necessary.

Common error: using HEALTHCHECK NONE unintentionally. If you inherit from a base image with a healthcheck and you set HEALTHCHECK NONE in your Dockerfile, you disable the inherited healthcheck. That may be intentional, but sometimes developers copy a Dockerfile snippet without realizing it. Ensure that you actually want to disable it.

Example: Safe Database Healthcheck

For a PostgreSQL container, use pg_isready:

services:
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: example
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

pg_isready returns 0 if the server is accepting connections, non-zero otherwise.

Now that we have safe configurations, let us verify them.

Verification and Diagnostics

Once you have a healthcheck configured, you need to verify that it works as expected. This section covers how to observe healthcheck status, interpret output, and diagnose issues.

Viewing Container Health Status

Run docker ps to see a health status column:

docker ps --format "table {{.Names}}\t{{.Status}}"

Example output:

NAMES               STATUS
my-web              Up 12 seconds (health: starting)
my-db               Up 30 seconds (healthy)

A container can be in one of three health states:

• starting: initial state, or during start-period.
• healthy: healthcheck command returns 0.
• unhealthy: healthcheck command fails more than allowed retries.

Inspecting Healthcheck Logs

To see detailed healthcheck logs, use docker inspect:

docker inspect --format='{{json .State.Health}}' my-web | jq

Example output:

{
  "Status": "unhealthy",
  "FailingStreak": 1,
  "Log": [
    {
      "Start": "2023-09-01T12:00:00.000Z",
      "End": "2023-09-01T12:00:00.500Z",
      "ExitCode": 1,
      "Output": "curl: (7) Failed to connect to localhost port 80: Connection refused\n"
    }
  ]
}

This tells you exactly what went wrong. In this case, the web server was not ready when the healthcheck ran. The fix is to set an appropriate start_period to allow time for startup.

Testing Healthcheck Command Manually

You can test the healthcheck command by running it inside a running container:

docker exec my-web curl -f http://localhost/

If it returns exit code 0 and prints the page, the command works. If it fails, you can debug the command itself.

Expected Results of a Correct Healthcheck

For a correctly configured healthcheck, you should see:

• docker ps shows (healthy) after the start period.
• docker inspect shows "Status": "healthy" and "FailingStreak": 0.
• Log entries have ExitCode: 0.

Verifying Healthcheck in Compose

Compose includes health status in docker compose ps:

docker compose ps

Example:

NAME                COMMAND                  SERVICE             STATUS              PORTS
myproject-web-1     "/docker-entrypoint...."   web                 running (healthy)
myproject-db-1      "docker-entrypoint.s..."   db                  running (healthy)

Now let us look at failure modes and how to recover from them.

Failure Modes and Recovery

Healthchecks can fail for many reasons: incorrect command, missing dependencies, network issues, application bugs, or misconfigured parameters. This section categorizes common failure modes and provides recovery steps.

Failure Mode 1: Healthcheck Command Missing or Incorrect

Symptom: Healthcheck logs show exit code 127 (command not found) or 126 (command not executable). Cause: The specified command or its interpreter is not installed in the image, or the path is wrong. Recovery:

• Identify the missing tool: docker exec <container> which <command> (if which available) or attempt to run it.
• Either change the healthcheck to use a tool that exists in the image, or add the tool to the image via Dockerfile (e.g., RUN apt-get update && apt-get install -y curl).
• Rebuild the image and redeploy.

Example: Suppose your image is based on alpine and your healthcheck uses wget, but wget is not installed. Add RUN apk add --no-cache wget to the Dockerfile.

Failure Mode 2: Healthcheck Times Out

Symptom: Healthcheck log shows exit code 124 (timeout) or the status remains starting for a long time. Cause: The healthcheck command takes longer than --timeout to complete. This can happen if the endpoint is slow, or the command hangs. Recovery:

• Increase the --timeout value, but also investigate why the command is slow.
• Ensure the command has bounded execution. For example, use curl with a timeout: curl --max-time 5 -f http://localhost/.
• Check application performance; maybe the endpoint is legitimately slow.

Example: Original: HEALTHCHECK --interval=30s --timeout=3s CMD curl -f http://localhost/slow-endpoint The endpoint takes 10 seconds to respond, so healthcheck times out. Change to --timeout=15s or use curl --max-time 10.

Failure Mode 3: Healthcheck Flapping Between Healthy and Unhealthy

Symptom: docker inspect shows alternating healthy/unhealthy status; FailingStreak resets. Cause: The healthcheck command is intermittent, possibly due to race conditions, resource limits, or network instability. Recovery:

• Examine the healthcheck logs to see if failures are clustered or random.
• Increase --interval to reduce load, or --retries to tolerate transient failures.
• Fix the underlying issue: e.g., ensure the application has enough memory, or the healthcheck endpoint is not doing heavy work.

Example: A healthcheck that runs ps aux | grep myapp may fail if the process list is momentarily inconsistent. Use a more stable healthcheck like checking a file or a lightweight endpoint.

Failure Mode 4: Healthcheck Reports Healthy but Application is Unhealthy

Symptom: Container shows healthy, but the application is not serving requests correctly. Cause: The healthcheck command does not accurately reflect application health. For example, checking only that the process is running, or checking a static page that works even when the backend is down. Recovery:

• Design a healthcheck that exercises a critical path of your application.
• For a web app, use an endpoint that performs a database query and returns a success code.
• Update the healthcheck command accordingly.

Example: Instead of curl -f http://localhost/, use curl -f http://localhost/health, where /health checks the database connection and returns 200 only if all dependencies are OK.

Failure Mode 5: Healthcheck Interferes with Application Startup

Symptom: Container restarts repeatedly during startup because healthcheck fails before the application is ready. Cause: Missing or insufficient start_period. The default start period is 0s, so healthchecks begin immediately and count failures right away. Recovery:

• Set start_period to a generous value (e.g., 30s or more based on startup time).
• Monitor startup logs to determine how long the application takes to be ready.
• Increase start_period accordingly.

Example: A Java application takes 60 seconds to start. Set start_period=90s to avoid premature failures.

Failure Mode 6: Healthcheck Command Returns Non-Zero on Warning

Symptom: Healthcheck fails even though application seems fine; log shows exit code 1 with a warning message. Cause: Some commands return non-zero for warnings. For example, curl without -f returns 0 for HTTP 404, but with -f returns 22 for any HTTP error. Recovery:

• Adjust the command to accept 0 for expected non-fatal warnings.
• Use a custom script that handles expected exit codes.

Example: If your healthcheck endpoint is /health and it returns 404 when the service is not ready, using curl -f will mark it unhealthy immediately. You might prefer to allow 404 as a starting state and only fail on other errors:

curl -s -o /dev/null -w '%{http_code}' http://localhost/health | grep -q '200\|404'

Then exit 0 only if the code is 200 or 404.

Recovery General Steps

When a healthcheck fails, follow this recovery sequence:

1. Inspect healthcheck logs to get the exact output and exit code.
2. Manually run the command inside the container (docker exec) to reproduce.
3. Evaluate if the command is correct for the application state and dependencies.
4. Tune parameters: adjust interval, timeout, start_period, and retries.
5. Redeploy with the fixed configuration.
6. Monitor the container health to confirm stability.

Remember to always test changes locally before rolling out to production.

Common Pitfalls and How to Avoid Them

Even experienced Docker users can fall into traps when configuring healthchecks. Here are some pitfalls to watch out for, why they happen, and how to avoid or recover from them.

Pitfall 1: Using a healthcheck that depends on external services

Why it happens: You might configure a healthcheck that hits an external endpoint (e.g., a third-party API) to verify connectivity. If the external service goes down, your container gets marked unhealthy, even if your application is fine. How to avoid: Keep healthchecks focused on the container's own health. If you need to check external dependencies, do so within your application's own health endpoint, but avoid direct external calls in the healthcheck. How to recover: If you must check external connectivity, use a short timeout and generous retries, and ensure the healthcheck itself does not hang.

Pitfall 2: Healthcheck command produces too much output

Why it happens: Some healthcheck commands, like running a script that logs debug information, can generate a lot of output on every check. This can bloat logs and consume disk space. How to avoid: Ensure the healthcheck command is quiet. Use flags like -s for curl or redirect output to /dev/null where appropriate. For example:

curl -s -o /dev/null http://localhost/health

How to recover: If output is already excessive, modify the command to suppress output and redeploy.

Pitfall 3: Not setting a start_period for slow-starting applications

Why it happens: Many applications, especially those with initialization tasks or database migrations, take time to be ready. Without a start_period, Docker may mark them unhealthy before they finish starting, causing unnecessary restarts. How to avoid: Always set a start_period that is longer than the expected startup time. For applications with variable startup times, err on the side of longer start_period. How to recover: If you observe early failures, increase start_period and retries, and redeploy.

Pitfall 4: Healthcheck interval too short, causing high CPU usage

Why it happens: A very short --interval (e.g., 1 second) causes the healthcheck command to run frequently, potentially consuming significant CPU and memory, especially if the command is not lightweight. How to avoid: Choose an interval that balances responsiveness with resource usage. Typically 10-30 seconds is reasonable. For critical services, you might go lower, but monitor resource consumption. How to recover: If you notice high CPU, increase the interval and simplify the healthcheck command.

Pitfall 5: Ignoring healthcheck status in orchestration

Why it happens: In Docker Swarm or other orchestrators, health status can be used for rolling updates and self-healing. If you do not configure restart policies or update configurations properly, unhealthy containers might not be replaced. How to avoid: Ensure your orchestrator respects health status. For example, in Docker Compose with docker stack deploy, set restart_policy and update order to use healthchecks. How to recover: Review your deployment configuration and add appropriate restart policies.

Operations Checklist

Maintaining healthy containers requires ongoing attention. This checklist provides repeatable steps for operating and reviewing healthchecks in your environment. Assign a single owner for each task to ensure accountability. The suggested owner is an example; adjust to your team structure.

Daily Operations

• [ ] Check docker ps and docker compose ps for any containers not in healthy state. Owner: On-call engineer (e.g., Alex Johnson). Review daily at shift handoff.
• [ ] For unhealthy containers, inspect logs with docker inspect --format='{{json .State.Health}}' <container>. Owner: On-call engineer. Review immediately when unhealthy detected.
• [ ] Confirm that unhealthy containers are being restarted or replaced according to policy. Owner: On-call engineer. Review daily.

Weekly Review

• [ ] Review healthcheck logs for consistent failures or timeouts that may indicate performance degradation. Owner: Service owner (e.g., Priya Shah, Backend Lead). Review every Monday morning.
• [ ] Verify that healthcheck commands are still accurate for the application (e.g., endpoints still exist, dependencies included). Owner: Service owner. Review weekly.
• [ ] Ensure that start periods are appropriate for application startup times (especially after app updates). Owner: Service owner. Review after each deployment or weekly.

Configuration Changes

• [ ] When modifying Dockerfile or docker-compose.yml, test healthcheck locally with docker compose up -d and observe status. Owner: Developer making change. Test before commit.
• [ ] Use version control for configuration files and keep healthcheck definitions in the same repository as the application. Owner: Developer. Continuous.
• [ ] Document the expected healthcheck behavior for each service. Owner: Service owner. Update documentation after changes; review quarterly.

Capacity and Resource Checks

• [ ] Monitor resource usage; if healthchecks are causing high CPU or memory, consider increasing interval or simplifying the command. Owner: SRE (e.g., Marcus Lee). Review monthly.
• [ ] Ensure healthcheck commands do not overload the application (e.g., avoid expensive database queries in the healthcheck). Owner: Developer. Review during code review.

Security

• [ ] Avoid exposing sensitive information in healthcheck output; logs may be collected centrally. Owner: Security engineer (e.g., Sarah Chen). Review quarterly.
• [ ] Ensure healthcheck endpoints are not publicly accessible unless intended, and they do not reveal sensitive details. Owner: Security engineer. Review after any network changes.

Incident Response

• [ ] When a container becomes unhealthy, gather healthcheck logs immediately. Owner: Incident commander. During incident.
• [ ] Preserve the output of docker inspect for post-incident analysis. Owner: Incident commander. During incident.
• [ ] Before restarting, try to determine root cause; if safe, run the healthcheck command manually inside the container. Owner: Incident commander. During incident.

Automation

• [ ] Integrate healthcheck monitoring into your alerting system (e.g., use Docker health status in container metrics). Owner: Monitoring team lead (e.g., David Kim). Set up initially, review quarterly.
• [ ] Use orchestration tools that respect health status for rolling updates or self-healing (e.g., Docker swarm mode, Nomad, etc.) but note that standalone Docker does not auto-heal without restart policies; ensure restart policies are set appropriately. Owner: Platform engineer (e.g., Emily Zhang). Review after infrastructure changes.

Conclusion

Docker healthchecks are a powerful tool for maintaining application reliability, but they require careful configuration and monitoring. We have covered the fundamentals of healthcheck syntax, common errors such as missing commands, timeouts, flapping, and false positives, and provided safe configuration patterns for Dockerfile and Compose. We also looked at how to verify healthchecks using docker inspect and docker exec, and outlined recovery steps for various failure modes. Finally, we provided an operations checklist to keep your healthchecks effective over time.

To move forward:

• Review your existing containers and identify any without healthchecks; add them where applicable.
• For existing healthchecks, inspect logs to ensure they are working correctly; fix any errors using the patterns in this article.
• Establish a routine to monitor health status and review configurations periodically.

With a solid healthcheck strategy, you can detect problems early, reduce downtime, and ensure that your containers are always in a known good state. Start with a narrow, measurable healthcheck for each critical service, test locally, and iterate based on the diagnostics you observe. Happy containerizing!