Hardening Docker Multi-Stage Builds: A Practical Security Guide

Intro

Docker multi-stage builds let you compile an application in one container and copy only the runtime artifacts into a much smaller final image. The pattern reduces size and attack surface, but the security value depends on how carefully each stage is structured. A build that leaves credentials in a layer, runs the final container as root, or copies more than the compiled binary still creates risk.

This guide shows how to harden multi-stage builds with concrete examples: selecting base images, using BuildKit secret mounts, enforcing a non-root user, minimizing the final filesystem, pinning dependencies, and verifying the result. The commands and Dockerfile snippets are designed so you can apply them incrementally to an existing build, observe the result, and roll back if something breaks.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before changing any Dockerfile, know the versions and tools involved. Security features and syntax differ across Docker Engine, BuildKit, and the container runtime. Run these commands to capture the environment:

docker version --format '{{.Server.Version}}'    # Docker Engine version
docker buildx version                             # Buildx and BuildKit support
docker info --format '{{.SecurityOptions}}'       # seccomp, AppArmor, SELinux status

Example output from a typical workstation:

Docker Engine: 24.0.7
buildx: v0.12.0
Security Options: name=seccomp,profile=builtin name=apparmor

If BuildKit is not enabled by default, set it as an environment variable or in the Docker daemon configuration. Modern Docker versions (23 and later) enable BuildKit automatically. For older versions, run builds with DOCKER_BUILDKIT=1 docker build ... or add this to /etc/docker/daemon.json:

{
  "features": {
    "buildkit": true
  }
}

Then restart Docker and verify with docker buildx version. BuildKit is required for secret mounts and some cache features used later.

Also confirm the target deployment platform. Multi-architecture builds need docker buildx build --platform linux/amd64,linux/arm64. The base image must exist for those platforms; otherwise the build fails.

Accountability: The DevOps engineer responsible for the build pipeline should document the Docker Engine and BuildKit versions and revisit them on a monthly cadence or whenever a new Node LTS or Go release is adopted.

Safe Configuration Path

The safe path is to make one small change at a time and verify it before moving on. Start with a working multi-stage Dockerfile and improve it incrementally. A typical Node.js example:

# Before hardening
FROM node:18 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:18-slim
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY package*.json ./
RUN npm ci --omit=dev
CMD ["node", "dist/index.js"]

This image still contains the full Node.js runtime, runs as root, and includes unnecessary files in the final layer. Hardening proceeds in small steps, each with a verification command.

Step 1: Add a non-root user and set ownership.

FROM node:18-slim
RUN groupadd -r appuser && useradd -r -g appuser appuser
WORKDIR /app
COPY --from=build --chown=appuser:appuser /app/dist ./dist
COPY --chown=appuser:appuser package*.json ./
RUN npm ci --omit=dev && chown -R appuser:appuser /app
USER appuser
CMD ["node", "dist/index.js"]

Verify the user context inside the container:

docker run --rm myimage id
# Expected: uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)

If the application needs to bind to a port below 1024, use a higher port internally and map it on the host, or grant the capability explicitly: docker run --cap-add=NET_BIND_SERVICE ....

Step 2: Move to a distroless or minimal runtime image.

For Node.js, Google's distroless/nodejs18-debian12 includes only the runtime and CA certificates. The build stage still uses the full image.

FROM node:18 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM gcr.io/distroless/nodejs18-debian12
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
CMD ["dist/index.js"]

Note: with a distroless image there is no shell, so debugging requires docker run --entrypoint /nodejs/bin/node ... and inspecting via docker exec is limited. The final image no longer has apt, bash, or curl, reducing attack surface.

Verify the image has no package manager:

docker run --rm myimage which apt
# Expected: error, command not found

Step 3: Separate build and runtime dependencies.

In the build stage, install only what is needed to compile. If the application uses native modules, the build stage may need python3, make, and g++, while the runtime stage does not.

FROM node:18 AS build
RUN apt-get update && apt-get install -y --no-install-recommends python3 make g++
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM gcr.io/distroless/nodejs18-debian12
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
CMD ["dist/index.js"]

Step 4: Use secret mounts for build-time credentials.

Never embed secrets like npm tokens or SSH keys in the image. BuildKit secret mounts make the secret available only for one RUN command without persisting it in a layer.

# syntax=docker/dockerfile:1.4
FROM node:18 AS build
RUN --mount=type=secret,id=npm_token \
    export NPM_TOKEN=$(cat /run/secrets/npm_token) && \
    npm config set //registry.npmjs.org/:_authToken=$NPM_TOKEN && \
    npm ci
# rest of build

Pass the secret at build time from a file or environment variable:

docker build --secret id=npm_token,src=./npm_token.txt -t myimage .

The file npm_token.txt is not copied into the image. After the RUN instruction, the secret is unmounted and gone.

Step 5: Reduce the final image content.

Only copy the compiled application and its runtime dependencies. Exclude test files, source maps, and build tools. For a Go binary, this is straightforward:

FROM golang:1.21 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o /out/app .

FROM gcr.io/distroless/static-debian12
COPY --from=build /out/app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]

The final image contains only the static binary and CA certificates. Verify the size:

docker images myimage
# REPOSITORY   TAG       IMAGE ID       CREATED          SIZE
# myimage      latest    a1b2c3d4e5f6   10 seconds ago   12.5MB

Compared to the original node:18 full image (around 950 MB), the reduction is dramatic.

Step 6: Pin base image digests.

Tags like node:18 move. Pin by digest to ensure reproducibility and protect against a compromised tag. Find the digest of a base image:

docker pull node:18-slim
 docker images --digests node
# REPOSITORY   TAG        DIGEST                                                                    IMAGE ID       CREATED         SIZE
# node        18-slim    sha256:2f7f7e9b3bb4d9f1e4f2d8b3e2a8d9c3c6f4a9d0a1d5c6e9f2c1e9e5f7b2c1   6f7f8c8d9e0a   2 weeks ago     177MB

Use the digest in the FROM line:

FROM node:18-slim@sha256:2f7f7e9b3bb4d9f1e4f2d8b3e2a8d9c3c6f4a9d0a1d5c6e9f2c1e9e5f7b2c1 AS build

Step 7: Add a HEALTHCHECK.

A healthcheck helps the orchestrator know if the container is functioning correctly.

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD ["/app/healthcheck"]

For a Node.js app without a separate binary, use node -e "require('http').get('http://localhost:3000/health', r => process.exit(r.statusCode === 200 ? 0 : 1))". Test locally with docker run --health-cmd ... and docker inspect --format='{{.State.Health.Status}}' <container>.

Verification and Diagnostics

After each hardening step, verify the image behaves correctly and has the expected security properties.

Check 1: Image history and layers.

Never expose secrets in image layers. Use docker history to scan for suspicious content.

docker history --no-trunc myimage

Look for lines that include ENV SECRET=... or RUN echo secret > file. If you used a secret mount, the secret should not appear in any layer. To be certain, scan the image with a tool like trivy:

trivy image --severity HIGH,CRITICAL myimage

Check 2: File system contents.

Run a temporary container and list the filesystem:

docker run --rm --entrypoint ls myimage -la /app

Expected output: only the application files, no .git, no node_modules with dev dependencies, no Dockerfile.

Check 3: Running user.

docker run --rm --entrypoint id myimage
# Expected: uid=65532(nonroot) gid=65532(nonroot) groups=65532(nonroot)

Check 4: Capabilities and security options.

docker run --rm --cap-drop ALL --security-opt no-new-privileges myimage

If the application needs specific capabilities, add them back one by one and document why.

Check 5: Network exposure.

Ensure the container listens only on the intended port. With docker run -p 127.0.0.1:3000:3000 ..., the service is not exposed to the host's external interfaces.

Check 6: Runtime security features.

Run with a read-only root filesystem and a non-root user:

docker run --rm --read-only --tmpfs /tmp --user 1000:1000 myimage

If the application writes to /tmp, the tmpfs mount allows it. For other writable locations like /var/run, add additional --tmpfs mounts.

Failure Modes and Recovery

Even with hardened builds, failures occur. Common failure modes and recovery steps:

Failure: Build fails with secret not found.

Cause: secret mount ID not passed with --secret or environment variable not set.
Recovery: Ensure the secret file exists and is readable. For CI systems, use the secret manager to inject the file. Run the build with --progress=plain to see the exact error.

Failure: Runtime error exec: "node": executable file not found in $PATH when switching to distroless.

Cause: distroless images do not include the node binary in $PATH; the binary is at /nodejs/bin/node.
Recovery: Use the full path in CMD or set ENV PATH=/nodejs/bin:$PATH in the Dockerfile.

Failure: Application cannot bind to low port.

Cause: non-root user lacks NET_BIND_SERVICE capability.
Recovery: Use a port above 1024 or add --cap-add=NET_BIND_SERVICE (not recommended for production). Better: configure reverse proxy to forward to high port.

Failure: Healthcheck fails in orchestration.

Cause: healthcheck command missing from distroless image or wrong endpoint.
Recovery: Use a compiled healthcheck binary or node -e with the full path. Test locally before deploying.

Failure: Image size not reduced.

Cause: copy from build stage includes unnecessary files or wrong base image.
Recovery: Inspect with docker history --human myimage to see layer sizes. Adjust COPY commands to target only the needed directories.

Failure: Container crashes with Permission denied writing to a mounted volume.

Cause: non-root user has no write permission on the named volume's initial directory.
Recovery: Pre-create the directory in the Dockerfile with RUN mkdir -p /app/data && chown appuser:appuser /app/data. For bind mounts from host, adjust host directory permissions or use an init container.

Common Pitfalls

Several pitfalls undermine security despite using multi-stage builds.

Pitfall 1: Copying the entire build context into the final image.

Instead of COPY --from=build /app /app, copy only the necessary artifacts. This keeps source code, tests, and build metadata out of the production image.

Pitfall 2: Using latest or mutable tags for base images.

Unpinned tags lead to unreproducible builds and potential supply chain attacks. Always pin by digest or at least a specific version tag.

Pitfall 3: Leaving credentials in environment variables.

Build-time secrets should use secret mounts, not ENV or ARG. ARG values are persisted in image history and can be extracted.

Pitfall 4: Running as root.

The default user in most base images is root. Running as root inside the container increases the risk of container breakout. Always switch to a non-root user.

Pitfall 5: Not dropping capabilities.

By default, containers have a set of capabilities. Dropping all and adding back only what is needed reduces the kernel attack surface. Use --cap-drop ALL and document required capabilities.

Pitfall 6: Missing healthchecks and liveness probes.

Without healthchecks, orchestration cannot restart unhealthy containers. Add a meaningful healthcheck that tests the application, not just the process.

Pitfall 7: Ignoring image vulnerabilities.

Regularly scan images with vulnerability scanners and update base images. Set up CI to break the build on critical vulnerabilities.

Pitfall 8: Not using multi-stage builds for static languages.

Even Go or Rust binaries benefit from multi-stage builds to separate the compiler toolchain from the runtime. The final image from scratch or distroless/static contains only the binary.

Accountability: The application team lead is accountable for each pitfall check at the time of Dockerfile pull request review. The security engineer revisits the vulnerability scan policy quarterly.

Operations Checklist

Use this checklist before promoting a hardened image to production.

Base image: pinned by digest? FROM node:18-slim@sha256:...
Non-root user: USER directive present and verified with docker run --rm --entrypoint id?
Secrets: no ENV or ARG with secret material; secret mounts used for build.
Final image content: only runtime artifacts copied; no source code, tests, or build tools.
Capabilities: drop all by default, add back only required ones (documented).
Healthcheck: present and tested locally; passes in orchestrator.
Read-only filesystem: container runs with --read-only and necessary tmpfs mounts.
Vulnerability scan: no high or critical vulnerabilities in the final image.
Image size: reasonable for the technology (e.g., <50 MB for Go binary, <200 MB for Node.js app).
Build reproducibility: same source and Dockerfile produce identical digest (docker build --iidfile and compare).
Dockerfile review: approved by security or DevOps reviewer.
Runtime configuration: orchestration manifest applies resource limits, security context, and network policies.
Rollback plan: previous known-good image tag or digest documented.

Owner: The platform engineer is the single accountable owner for the checklist completion on each release. The checklist is enforced in CI as a gating job and reviewed weekly during the release readiness meeting.

Conclusion

Docker multi-stage builds provide a strong foundation for container security, but they are not automatically secure. Hardening requires deliberate choices: pinned minimal base images, non-root users, secret mounts, capability dropping, and rigorous verification. Each change should be applied incrementally with observation and rollback options.

Start with one low-risk improvement, such as pinning the base image digest or adding a non-root user. Record the current image size and vulnerability report, apply the change, and compare. Over time, the build pipeline becomes a repeatable process that produces smaller, more secure images.

Security is a continuous activity. Regularly revisit base images, vulnerability findings, and runtime policies. The practices in this guide form a baseline that can be adapted to your specific application stack and deployment environment.