## Intro Docker multi-stage builds are the standard way to produce small, secure, production-ready images. The basic idea is simple: use one stage to compile or prepare the application, then copy only the necessary artifacts into a final image. But advanced use goes far beyond a simple two-stage Dockerfile. This article dives into the techniques that separate a production-grade pipeline from a hobby project: precise cache control, conditional builds with BuildKit, building only a specific target stage, sharing data between stages with `--mount`, and hardening the final image. Each concept is paired with a concrete example you can run today. By the end, you will know how to keep builds fast, avoid leaking secrets, and produce images that are easier to secure and maintain. ## Why Multi-Stage Builds Matter A single-stage Dockerfile for a Go application might look like this: ```dockerfile FROM golang:1.22 WORKDIR /app COPY . . RUN go build -o myapp . CMD ["./myapp"] ``` The resulting image includes the entire Go toolchain, source code, and build cache, often reaching over 800 MB. In production, that means slower pulls, a larger attack surface, and wasted disk space. Multi-stage builds solve this by separating the build environment from the runtime environment. The same application can be built in under 20 MB using a scratch or alpine base: ```dockerfile # Stage 1: build FROM golang:1.22 AS builder WORKDIR /app COPY go.mod go.sum ./ RUN go mod download COPY . . RUN CGO_ENABLED=0 go build -o myapp . # Stage 2: run FROM alpine:3.19 RUN apk --no-cache add ca-certificates WORKDIR /root/ COPY --from=builder /app/myapp . CMD ["./myapp"] ``` This pattern is well-known. But to use it effectively in a real CI/CD pipeline, you need to understand the finer points. ## Advanced Caching Strategies ### Layer Caching and Order of Instructions Docker caches layers based on the instruction and its inputs. Changing any instruction invalidates that layer and all subsequent layers. Therefore, ordering matters enormously. In the Go example above, we copy `go.mod` and `go.sum` before copying the rest of the source. This way, dependency downloads are cached separately from application code changes. If you only change `main.go`, the `go mod download` layer is reused, saving network time. A common mistake is to copy everything at once: ```dockerfile COPY . . RUN go mod download RUN go build -o myapp . ``` Here, any change to any file invalidates the copy layer, causing `go mod download` to run again. Always separate dependency installation from source copying. ### Using BuildKit Cache Mounts BuildKit (enabled by default in Docker 23.0+ with the Docker driver, or via `DOCKER_BUILDKIT=1`) introduces cache mounts that persist between builds without storing data in the final image. They are perfect for package managers, compilers, and test artifacts. For example, to cache Go modules during a build: ```dockerfile # syntax=docker/dockerfile:1.7 FROM golang:1.22 AS builder WORKDIR /app COPY go.mod go.sum ./ RUN --mount=type=cache,target=/go/pkg/mod \ go mod download COPY . . RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ go build -o myapp . ``` The `--mount=type=cache` directive tells BuildKit to mount a persistent cache directory at the specified path during this RUN instruction. The cache survives across builds (as long as the mount ID is stable) and is not included in the final image. This can dramatically speed up builds. For Node.js, you might cache `node_modules` or the npm cache: ```dockerfile FROM node:20 AS build WORKDIR /app COPY package*.json ./ RUN --mount=type=cache,target=/root/.npm \ npm ci COPY . . RUN npm run build ``` Here, `/root/.npm` caches the npm package cache, so re-running `npm ci` does not re-download packages from the registry every time. For apt-based images, you can cache `/var/cache/apt` and `/var/lib/apt`: ```dockerfile FROM debian:12 AS build RUN --mount=type=cache,target=/var/cache/apt \ --mount=type=cache,target=/var/lib/apt \ apt-get update && apt-get install -y --no-install-recommends \ build-essential ``` This avoids re-downloading package lists on every build. ### Secrets Management During Build Never bake secrets into an image. Even if you later delete them, they remain in intermediate layers. BuildKit provides secret mounts to make secrets available only during a specific RUN instruction. Example: using an SSH key to clone a private repository: ```dockerfile # syntax=docker/dockerfile:1.7 FROM alpine:3.19 RUN apk add --no-cache git openssh-client RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts RUN --mount=type=ssh git clone git@github.com:myorg/private-repo.git /src ``` Build with: ```bash docker build --ssh default -t myapp . ``` The SSH agent socket is mounted only for the `git clone` command, and the private key never ends up in the image. Similarly, you can mount a secret file: ```dockerfile RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret ``` Build with: ```bash docker build --secret id=mysecret,src=./secret.txt -t myapp . ``` This is far safer than using `ARG` for secrets, because `ARG` values are visible in build history. ## Conditional Builds with BuildKit With BuildKit, you can use inline RUN mounts and heredocs, and more importantly, you can conditionally include stages using build args. This is useful for creating debug builds, selecting different CPU architectures, or toggling features. Consider an image that can be built either as a development image (with debug symbols and tools) or a production image (minimal). ```dockerfile # syntax=docker/dockerfile:1.7 FROM alpine:3.19 AS base RUN apk add --no-cache ca-certificates WORKDIR /app COPY --from=builder /app/myapp . FROM base AS production CMD ["./myapp"] FROM base AS development RUN apk add --no-cache curl bash CMD ["sh"] ``` Then build selectively: ```bash docker build --target production -t myapp:prod . docker build --target development -t myapp:dev . ``` You can also use build args to choose between alternatives within a stage: ```dockerfile ARG VERSION=latest FROM base AS builder RUN if [ "$VERSION" = "debug" ]; then \ go build -gcflags="all=-N -l" -o myapp .; \ else \ go build -o myapp .; \ fi ``` While `if` statements work, they can become unwieldy for complex conditions. A cleaner approach is to define separate stages and select with `--target`. ## Building Only a Specific Stage In a large Dockerfile with many stages, you often want to build just one. For example, you might have a `test` stage that runs unit tests, and you want to run it separately from the final image build. ```dockerfile # syntax=docker/dockerfile:1.7 FROM golang:1.22 AS base WORKDIR /app COPY go.mod go.sum ./ RUN go mod download COPY . . FROM base AS test RUN go test ./... FROM base AS build RUN CGO_ENABLED=0 go build -o myapp . FROM alpine:3.19 AS final COPY --from=build /app/myapp /usr/local/bin/myapp CMD ["myapp"] ``` You can run just the tests with: ```bash docker build --target test . ``` This avoids building the final image when you only need test results. In CI, this can be used in a pipeline to run tests before proceeding to build. Note: because `test` depends on `base`, Docker will still build `base` first, but it will not build `build` or `final`. ## Sharing Data Between Stages with `--mount` Sometimes you need to pass more than just a few files between stages. BuildKit's `--mount=type=bind` and `--mount=type=cache` offer more flexibility than `COPY --from`. For example, suppose your build stage generates multiple artifacts (binaries, configs, static assets) that you want to copy into the final image, but you also want to run a post-processing step. You could copy each artifact individually with `COPY --from`, but that can be verbose. An alternative is to use a named context or a bind mount from a previous stage. However, `COPY --from` remains the simplest for most cases. The power of `--mount` shines when you want to share a directory during the build without copying files into a layer that might become stale. Consider a monorepo where the build stage compiles a frontend and a backend, and they need to share common code. You can bind-mount the source directory from the build context into both stages: ```dockerfile FROM node:20 AS frontend-build WORKDIR /app RUN --mount=type=bind,source=.,target=/app \ npm install && npm run build:frontend FROM golang:1.22 AS backend-build WORKDIR /app RUN --mount=type=bind,source=.,target=/app \ go build -o server ./backend ``` But this approach re-mounts the entire context for each stage, which may not be efficient. It is generally better to use `COPY` for specific files and rely on layer caching. One special use case is `--mount=type=cache` shared between stages. For example, you might want to reuse a downloaded dependency cache across multiple build stages: ```dockerfile # syntax=docker/dockerfile:1.7 FROM golang:1.22 AS build WORKDIR /app COPY go.mod go.sum ./ RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ go mod download COPY . . RUN --mount=type=cache,target=/go/pkg/mod \ --mount=type=cache,target=/root/.cache/go-build \ go build -o myapp . ``` The cache mount is shared across repeated builds, and even across different stages if they use the same target path and ID. But note that cache mounts are not shared between concurrent builds by default. ## Squashing Layers and Reducing Image Size While multi-stage builds already produce smaller images, you can go further with layer squashing and `COPY --link`. `COPY --link` (requires BuildKit) copies files from another stage without creating a dependency on the previous layers. This can allow the copy to be performed in parallel and reduces the number of layers in some cases. ```dockerfile # syntax=docker/dockerfile:1.7 FROM alpine:3.19 AS final COPY --link --from=build /app/myapp /usr/local/bin/myapp ``` This is especially useful when you have many small files to copy; instead of one layer per COPY, they can be combined. To further reduce size, consider using `scratch` as the final base if your application is statically linked or does not need any OS utilities. For example, a Go binary built with `CGO_ENABLED=0` can run on `scratch`: ```dockerfile FROM scratch COPY --from=build /app/myapp /myapp ENTRYPOINT ["/myapp"] ``` This produces an image that contains only the binary, often under 10 MB. However, be aware that `scratch` has no shell, no CA certificates (unless copied), and no timezone data. You must include any needed files explicitly. If you need CA certificates, copy them from a builder stage: ```dockerfile FROM alpine:3.19 AS certs RUN apk add --no-cache ca-certificates FROM scratch COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ COPY --from=build /app/myapp /myapp ENTRYPOINT ["/myapp"] ``` This keeps the image tiny while allowing HTTPS calls. ## Security Hardening Multi-stage builds help security by separating build tools from runtime. But there are more steps you can take: 1. **Use specific base image versions, not `latest`.** Pinning to `alpine:3.19` or `debian:12-slim` ensures reproducibility and reduces surprises. 2. **Create a non-root user in the final stage.** Avoid running containers as root. ```dockerfile FROM alpine:3.19 RUN addgroup -S app && adduser -S app -G app USER app COPY --from=build --chown=app:app /app/myapp /usr/local/bin/myapp CMD ["myapp"] ``` 3. **Remove unnecessary setuid/setgid permissions** and use `--no-install-recommends` for apt. 4. **Use a scanner** like Trivy or Clair to identify vulnerabilities in the final image. 5. **Set `LABEL` for metadata** but do not include sensitive info. Another powerful technique is to use BuildKit's `--sbom` and `--provenance` flags to generate attestations, but that is beyond the scope of this article. ## Common Pitfalls and How to Avoid Them 1. **Not ordering layers for cache efficiency.** Mistake: copying all source before installing dependencies. Impact: cache invalidates on every code change. Fix: copy dependency files first. 2. **Leaking secrets via build args.** Mistake: `ARG mysecret` and then using it in a RUN. Fix: use secret mounts or environment variables at runtime. 3. **Using `latest` base images.** Mistake: unpredictability across builds. Fix: pin to a digest or specific version. 4. **Building images as root and running as root.** Mistake: container compromise gives root on host. Fix: create non-root user. 5. **Forgetting to clean up in the final stage.** Mistake: copying unnecessary files or including package manager caches. Fix: use `--no-cache` and inspect image size. 6. **Not using BuildKit.** Mistake: missing out on cache mounts, secret mounts, and parallel builds. Fix: ensure `DOCKER_BUILDKIT=1` or Docker 23+ default. 7. **Ignoring layer count.** Mistake: too many layers can bloat image metadata and slow pulls. Fix: combine RUN commands where sensible and use multi-stage copies. 8. **Using `COPY . .` without `.dockerignore`.** Mistake: sending `.git`, local secrets, or large files to the daemon. Fix: create a `.dockerignore` file. Example `.dockerignore`: ```text .git .env *.md node_modules ``` 9. **Not leveraging `--target` for testing.** Mistake: rebuilding entire image just to run tests. Fix: define a test stage and use `docker build --target test .` 10. **Assuming multi-stage automatically shrinks image.** Mistake: if the final stage includes unnecessary packages, it remains large. Fix: regularly check `docker images` and optimize. ## Real-World Example: Node.js Frontend with Backend API Let's walk through a complete Dockerfile for a typical web application: a Node.js frontend that gets built to static files, and a Go backend serving the API. We want a final image that serves both the API and the static frontend files. ```dockerfile # syntax=docker/dockerfile:1.7 # ---- Frontend Build ---- FROM node:20-alpine AS frontend-build WORKDIR /app/frontend COPY frontend/package*.json ./ RUN npm ci COPY frontend/ . RUN npm run build # ---- Backend Build ---- FROM golang:1.22-alpine AS backend-build WORKDIR /app/backend COPY backend/go.mod backend/go.sum ./ RUN go mod download COPY backend/ . RUN CGO_ENABLED=0 go build -o server . # ---- Final Stage ---- FROM alpine:3.19 RUN apk add --no-cache ca-certificates WORKDIR /app COPY --from=backend-build /app/backend/server ./server COPY --from=frontend-build /app/frontend/dist ./static EXPOSE 8080 USER nobody CMD ["./server"] ``` In this example, the frontend and backend are built in parallel (BuildKit can run independent stages concurrently), and only the compiled binary and static files are copied into the final image. The result is a small image (likely under 30 MB) that runs as a non-root user. To build and run: ```bash docker build -t myapp . docker run -p 8080:8080 myapp ``` The backend server must be configured to serve the static files from `/app/static`. For a Go server, that might mean embedding the static files or serving via `http.FileServer`. ## Operations Checklist for Multi-Stage Builds Before pushing an image to production, verify the following: - [ ] Base images are pinned by digest or specific version (e.g., `alpine:3.19@sha256:...`). - [ ] `.dockerignore` excludes `.git`, secrets, logs, and build artifacts. - [ ] No secrets are passed via `--build-arg`; secret mounts or runtime env vars are used instead. - [ ] The final image runs as a non-root user (`USER` instruction set). - [ ] The image contains only necessary files (check with `docker run --rm -it myimage sh` and `ls`). - [ ] Caches are utilized: dependency layers are separated from source changes. - [ ] BuildKit is enabled and `--mount=type=cache` is used where appropriate. - [ ] If using `scratch`, required CA certificates and timezone data are included. - [ ] Layer count is reasonable (combine related RUN commands). - [ ] Image size measured and compared to previous builds (`docker images myapp`). - [ ] A security scan has been run (e.g., `trivy image myapp`). - [ ] The build is reproducible: building in a clean environment yields the same image ID (consider using `--provenance` and `--sbom`). **Accountability**: The DevOps engineer (e.g., Priya Shah, Engineering Lead) is the single accountable owner for the multi-stage build definition and its review. The build process should be reviewed at least quarterly or whenever a base image changes, a new dependency is added, or a security advisory affects the stack. ## Failure Modes and Recovery Even with careful planning, issues arise. Here are common failures and how to recover: 1. **Build fails due to missing secret or file.** Symptom: `ERROR: failed to solve: ... not found`. Recovery: ensure the secret file exists and is passed correctly (`--secret` or `--ssh`). Check that the path inside the container is correct. 2. **Cache not being utilized.** Symptom: build always re-downloads dependencies. Recovery: verify the instruction order and that cache mounts are properly defined. Use `docker build --progress=plain` to see cache hits. Ensure BuildKit is active. 3. **Image size unexpectedly large.** Symptom: `docker images` shows size > expected. Recovery: run `docker history ` to see layer sizes. Identify large layers; remove unnecessary files; consider `scratch` or `alpine` final stage; use `--no-install-recommends`. 4. **Container crashes with `not found` for CA certs or timezone.** Recovery: copy CA certs from an alpine stage as shown earlier; set timezone environment variable or copy tzdata. 5. **Permission denied for non-root user.** Symptom: application cannot write to a directory. Recovery: ensure the directory is owned by the same user/group using `--chown` in COPY or `RUN chown`. 6. **Secrets leaked in image layers.** Recovery: immediately rotate the secret. Rebuild the image with secret mounts. Consider using multi-stage to ensure secrets are not in final layers. Use `docker scan` or `docker history` to detect if needed. 7. **Layer caching causes stale dependencies.** Symptom: old version of dependency is used because layer cache is valid. Recovery: bust the cache by changing the relevant file (e.g., update `go.sum` or `package-lock.json`) or use `--no-cache`. 8. **BuildKit cache mount fills disk.** Symptom: build host disk usage grows over time. Recovery: periodically prune cache with `docker builder prune` and set cache limits. ## Conclusion Multi-stage builds are a cornerstone of efficient Docker usage, but their advanced features can make a significant difference in build speed, security, and maintainability. By applying the techniques in this article—cache mounts, secret mounts, selective stage targeting, and proper user handling—you can build production images that are small, fast to build, and secure. The key is to treat the Dockerfile as a critical piece of infrastructure: version it, review it, and optimize it continuously. Start with one improvement: enable BuildKit, add a `.dockerignore`, use cache mounts for your package manager, and introduce a non-root user. Then measure the impact and iterate. With these practices, your containers will be lean, your builds rapid, and your attack surface reduced.