Docker health checks and restart policies for reliable containers: practical implementation guide

If you rely on Docker to run customer-facing services, reliability is your brand. Two features carry a lot of that weight: health checks and restart policies. Health checks tell Docker when a service is actually ready or broken. Restart policies tell Docker what to do when a container stops or crashes. Together with dependency timing in Docker Compose, they help you avoid flaky startups, silent failures, and midnight surprises. This guide shows exactly how to implement and test them, using short examples you can run on your laptop.

Workflow Overview

Add a fast, deterministic HEALTHCHECK to each service.
Mirror the same probe in Docker Compose for local stacks.
Select a restart policy per service based on failure behavior.
Use depends_on with service_healthy for startup ordering.
Test common failure modes locally (slow start, dependency down, probe failure).
Roll out incrementally with logs and metrics you can inspect.

Health check fundamentals

Health checks answer a simple question: is the container doing useful work right now?

Key principles:

Keep it fast: under a few seconds. Slow checks become the problem.
Check from inside the container: target localhost, not external hosts.
Use exit codes: 0 is healthy, non-zero is unhealthy.
Add a warmup: use start_period to avoid false negatives on cold start.

Dockerfile example (Node.js HTTP service):

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000
# Health check: wait a bit for warmup, then probe /health
HEALTHCHECK --interval=30s --timeout=3s --start-period=30s --retries=3 \
  CMD wget -qO- http://localhost:3000/health || exit 1
CMD ["node", "server.js"]

Compose health check for the same service:

services:
  web:
    build: .
    ports:
      - "3000:3000"
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:3000/health || exit 1"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 30s

Inspect health state:

# Show health status quickly
docker ps --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

# Inspect detailed health history
docker inspect --format='{{json .State.Health}}' web | jq .

# Stream health events
docker events --filter event=health_status

Restart policies explained

Restart policies control what Docker does when a container stops. They do not automatically restart a container just because it is unhealthy. Restarts are triggered by exit, not by health status.

Common options:

no: never restart (default if not specified).
on-failure[:max-retries]: restart on non-zero exit codes, optionally capped.
always: always restart when the container stops for any reason.
unless-stopped: restart like always, except if you manually stop it.

Compose examples:

services:
  worker:
    image: myorg/worker:1.2
    restart: "on-failure:5"  # retry at most 5 times on crash

  cron:
    image: myorg/cron:1.0
    restart: unless-stopped   # keep it running unless user stops it

  api:
    build: ./api
    restart: always           # bring it back after host reboots

Tips:

Choose on-failure for short-lived jobs that should retry.
Choose always or unless-stopped for long-running services.
If a container is stuck unhealthy but still running, it will not be restarted by policy alone. Investigate the cause or design the app to exit on fatal errors.

Compose health checks and timing

depends_on with health conditions lets you start services in a sensible order. It does not keep them in lockstep forever; it only gates startup.

Example: API waits for Postgres to be healthy.

version: "3.9"
services:
  db:
    image: postgres:15-alpine
    environment:
      POSTGRES_PASSWORD: example
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 20s

  api:
    build: ./api
    depends_on:
      db:
        condition: service_healthy
    environment:
      DATABASE_URL: postgres://postgres: example@db:5432/postgres
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8080/health || exit 1"]
      interval: 30s
      timeout: 3s
      retries: 3
      start_period: 30s

What this gives you:

db will report healthy only after pg_isready passes consistently.
api will not start until db is healthy, reducing boot-time failures.
If db goes down later, api stays up; handle reconnection in the app.

Failure modes and troubleshooting

Common issues and how to handle them:

Slow cold start makes the check fail early:
Increase start_period to cover warmup.
Use fewer retries with a slightly longer interval to reduce flapping.

Health check flaps under load:
Keep checks lightweight. Avoid full dependency calls when possible.
Check an internal readiness endpoint, not the home page.

Unhealthy but not restarting:
Remember: restart policies act on exit. If the service is beyond recovery, consider having the process exit non-zero so the policy can act.

Network or DNS hiccups inside the container:
Target localhost or the service container name, not external domains.
For self-checks, prefer 127.0.0.1 over hostnames.

Missing tools in minimal images:
Alpine often has wget via BusyBox; curl may not be installed.
If you need curl, add it explicitly (e.g., RUN apk add --no-cache curl) and measure image size impact.

Checks that change state:
Make probes idempotent. They should not write to databases or mutate caches.

Long I/O in health checks:
Set timeout to a small value (1-5s) so a stuck probe fails fast.

Database readiness vs. liveness:
A DB TCP port being open is not the same as being ready. Use pg_isready or an equivalent for your DB.

Useful commands:

# See live health transitions
docker events --filter event=health_status

# Watch current state in a loop
watch -n2 'docker ps --format "table {{.Names}}\t{{.Status}}"'

# Force a failure test by pausing a dependency
docker pause db && sleep 20 && docker unpause db

# View recent health logs
docker inspect --format='{{json .State.Health.Log}}' api | jq '.[-5:]'

Local Pilot Plan

Start narrow, measure results, and keep it local first.

Goal: Add a health check and restart policy to one service, verify behavior in three scenarios, and capture simple pass/fail signals.

Steps:

Select one service that has a /health or similar endpoint.
Add HEALTHCHECK in the Dockerfile with start_period and small timeouts.
Mirror the check in docker-compose.yml.
Pick a restart policy that matches intent (e.g., always for an API).
Test locally:

Normal: docker compose up -d; confirm Status includes (healthy).
Dependency down: stop the dependency, confirm health turns to (unhealthy), then restore.
Crash: docker kill -s SIGKILL <container>; confirm it restarts under the policy.

Record timings: warmup time, average probe duration, false failures, and restarts.
Adjust start_period, interval, timeout, and retries to remove false negatives.

Commands:

# Bring up the stack
docker compose up -d --build

# Check health summary
docker compose ps

# Induce a crash
docker kill -s SIGKILL api

# Bring down and clean up
docker compose down -v

Exit criteria:

Health turns healthy within an expected warmup window.
Crash triggers a restart as designed.
An unhealthy state is visible in docker ps and logs, without noisy flapping.

Conclusion

Reliable containers pair fast, meaningful health checks with restart policies that match how each service should behave on failure. Use Compose health checks and depends_on to order startup, then test slow starts, crashes, and dependency loss locally. Start with a small pilot, measure outcomes, and roll the pattern across the stack. The result is simpler operations and fewer surprises.