## Intro Upgrading and migrating Docker multi-stage builds is not a single copy-paste operation. It is a sequence of observations, scoped changes, and verification steps that protect your build pipeline and the images it produces. This guide walks through the process from start to finish - from checking your current Docker and BuildKit versions to rolling back safely if something breaks. You will learn how to: - Inventory your current Docker environment and multi-stage build configuration - Identify compatibility requirements and choose the right upgrade version - Test changes in a controlled way before touching production - Validate that the new build produces a correct, smaller image - Recover from common failures without losing data or build history The examples use a realistic Node.js application, but the patterns apply to any language or framework. Commands are shown with expected output and failure signals so you can compare what you see against what should happen. ## Version and Environment Inventory Before changing anything, document the exact state of your Docker installation and the builds you are migrating. This gives you a baseline for comparison and a rollback point. ### Check the Docker Engine and CLI Run these commands and record the output: ```bash docker version ``` Expected output includes separate sections for Client and Server with version numbers. Example: ``` Client: Docker Engine - Community Version: 24.0.7 API version: 1.43 Go version: go1.20.10 Git commit: 311b9ff Built: Thu Oct 26 09:08:01 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.7 API version: 1.43 (minimum version 1.12) Go version: go1.20.10 Git commit: 311b9ff Built: Thu Oct 26 09:08:01 2023 OS/Arch: linux/amd64 Experimental: false ``` If the Server section is missing or shows an error, Docker daemon is not running. Start it with `sudo systemctl start docker` (Linux) or Docker Desktop (Windows/macOS). ### Check BuildKit Support Multi-stage builds use BuildKit for advanced features like cache mount and secrets. Check if BuildKit is enabled: ```bash docker buildx version ``` Expected output: ``` github.com/docker/buildx v0.12.0 4b6b4b8 ``` If you see `docker: 'buildx' is not a docker command`, install the buildx plugin or enable legacy builder. For most modern Docker versions (20.10+), BuildKit is the default builder. Verify with: ```bash docker buildx ls ``` Example output: ``` NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS default * docker default default running 20.10.24 linux/amd64, linux/arm64 desktop-linux docker desktop-linux desktop-linux running 23.0.6 linux/amd64, linux/arm64 ``` ### Identify Your Current Multi-Stage Dockerfile Examine the Dockerfile you plan to migrate. A typical multi-stage build for a Node.js app might look like this: ```dockerfile # Stage 1: build FROM node:18 AS builder WORKDIR /app COPY package*.json ./ RUN npm ci COPY . . RUN npm run build # Stage 2: production FROM node:18-alpine WORKDIR /app COPY --from=builder /app/dist ./dist COPY package*.json ./ RUN npm ci --only=production CMD ["node", "dist/server.js"] ``` Note the stage names (`builder`, `production`), base images, and file copies. These will be the focal points for upgrade changes. ### Pre-upgrade Data and Volume Check If your build process uses volumes for caching (e.g., npm or Maven caches), confirm where those are stored. A named volume like `npm-cache:/root/.npm` persists across builds and can be reused. A bind mount like `./.npm-cache:/root/.npm` depends on the host directory existing. To test persistence, run a build with caching enabled, then restart the Docker daemon and rebuild. Compare build times and check if the cache was reused. Example: ```bash docker buildx build --push -t myapp:test --build-arg BUILDKIT_INLINE_CACHE=1 . # Note the build time, then rebuild after restart time docker buildx build --push -t myapp:test --build-arg BUILDKIT_INLINE_CACHE=1 . ``` If the second build is nearly as fast as the first and logs show `CACHED`, your volume configuration is correct. If the second build re-runs all steps, the cache was not persisted, and you need to adjust volume mounts. ## Safe Configuration Path This section covers the actual migration steps: updating the Dockerfile, environment, and builder configuration in a way that can be rolled back. ### Choose the Target Version Consult the Docker and BuildKit release notes for breaking changes. For example, Docker Engine 23.0 deprecated the legacy builder and made BuildKit the default. If you are moving from an older version, plan to test BuildKit explicitly. Target version example: - Current: Docker 20.10.24, BuildKit 0.8.2 - Target: Docker 24.0.7, BuildKit 0.12.0 Check that your CI runner or production host can install the target version. For apt-based systems: ```bash sudo apt-get update sudo apt-get install docker-ce=5:24.0.7-1~ubuntu.22.04~jammy docker-ce-cli=5:24.0.7-1~ubuntu.22.04~jammy ``` Always pin the exact version to avoid accidental upgrades. ### Update the Dockerfile Stage Definitions Migrate syntax changes. For example, if you were using the legacy `FROM ... AS` format, it is unchanged. But if you were using old cache mount syntax, update it. Old syntax (before BuildKit 0.9): ```dockerfile RUN --mount=type=cache,target=/root/.npm npm ci ``` New syntax: ```dockerfile RUN --mount=type=cache,target=/root/.npm npm ci ``` (No change here, but ensure you are not using deprecated flags like `--stream` in `docker build`.) More important: if you are moving from a single-stage to multi-stage, split the build. For a Python app: Before: ```dockerfile FROM python:3.9 WORKDIR /app COPY . . RUN pip install -r requirements.txt CMD ["python", "app.py"] ``` After: ```dockerfile FROM python:3.9 AS builder WORKDIR /app COPY requirements.txt . RUN pip install --user -r requirements.txt FROM python:3.9-slim WORKDIR /app COPY --from=builder /root/.local /root/.local COPY . . ENV PATH=/root/.local/bin:$PATH CMD ["python", "app.py"] ``` This reduces image size by excluding build tools and caches. ### Adjust Build Arguments and Environment If your build uses `ARG` for versioned dependencies, update them to match the new base image. Example: ```dockerfile ARG NODE_VERSION=18 FROM node:${NODE_VERSION} AS builder ``` When migrating to Node 20, change the default: ```dockerfile ARG NODE_VERSION=20 ``` Then build with: ```bash docker build --build-arg NODE_VERSION=20 -t myapp:node20-test . ``` Do not hardcode versions inside the Dockerfile unless necessary; use ARG defaults and override at build time for flexibility. ### Test the New Build in Isolation Before replacing existing images, build with a different tag and run smoke tests. ```bash docker build -t myapp:upgrade-test . docker run -d --name upgrade-test -p 3001:3000 myapp:upgrade-test curl http://localhost:3001/health ``` Expected: `{"status":"ok"}`. Check logs: ```bash docker logs upgrade-test --tail 20 ``` If the application fails to start, inspect the container: ```bash docker inspect upgrade-test | jq '.[0].State' ``` Look for `"Running": false` and `"ExitCode": 1` with an error message. ### Run a Parallel Build and Compare Images Compare the old and new images for size and layers: ```bash docker images myapp ``` Example output: ``` REPOSITORY TAG IMAGE ID CREATED SIZE myapp old-prod 1a2b3c4d5e6f 2 weeks ago 450MB myapp upgrade-test a1b2c3d4e5f6 10 minutes ago 320MB ``` A successful multi-stage migration should reduce image size, sometimes by 50% or more. Use `docker history myapp:upgrade-test` to verify that only the final stage files are present. ## Verification and Diagnostics After upgrading, confirm that the new build is correct and the resulting container behaves as expected. ### Automated Build Verification Run the build with checks that fail fast on errors: ```bash docker build --progress=plain --no-cache -t myapp:verify . 2>&1 | tee build.log ``` Inspect `build.log` for any warnings or errors. Common issues: - `WARNING: No output specified for stage X` - a stage is unused or wrongly defined - `ERROR: failed to solve: failed to compute cache key` - file paths mismatch - `ERROR: executor failed running [/bin/sh -c npm run build]: exit code: 1` - build script failed Each of these requires a different fix: check stage naming, confirm COPY paths, or debug the build command. ### Runtime Verification Commands For the running container, run health checks: ```bash docker exec upgrade-test sh -c "curl -s http://localhost:3000/health" ``` Expected: `{"status":"ok"}`. Check that the container uses the expected base image layers: ```bash docker inspect upgrade-test | jq '.[0].Config.Image' ``` Expected: `myapp:upgrade-test` (or the base image digest). ### Performance and Resource Comparison Measure startup time and memory usage between old and new containers. ```bash time docker run --rm myapp:old-prod true time docker run --rm myapp:upgrade-test true ``` For memory: ```bash docker stats --no-stream upgrade-test ``` Record values. A multi-stage build should not increase resource usage; often it decreases because the final image is leaner. ### Validate Image Content Ensure the final image does not contain build-time tools or source code. Use a temporary container to explore: ```bash docker run --rm -it myapp:upgrade-test sh ``` Inside, run: ```bash ls -la /app ``` You should see only the compiled output and runtime dependencies. If you see build artifacts like `node_modules` with dev dependencies, the multi-stage copy is too broad. Fix by copying only the needed directories: ```dockerfile COPY --from=builder /app/dist ./dist COPY --from=builder /app/node_modules ./node_modules ``` Do not copy `/app` wholesale. ## Failure Modes and Recovery Even with careful planning, upgrades can fail. Here are common failure modes and how to recover. ### Build Failure Due to Missing BuildKit Features If the build fails with an error like: ``` ERROR: BuildKit is enabled but the buildx component is missing or broken ``` Or: ``` ERROR: failed to solve: rpc error: code = Unknown desc = failed to load cache key ``` This often happens when the Docker version is older than the Dockerfile features used. Verify BuildKit version and consider using the legacy builder as a temporary fallback: ```bash DOCKER_BUILDKIT=0 docker build -t myapp:legacy . ``` But do not rely on legacy mode long-term; upgrade Docker instead. ### Base Image Pull Failure The new base image may not exist or may be incompatible with your architecture. Error: ``` ERROR: failed to solve: node:20: not found ``` Check available tags: ```bash docker manifest inspect node:20 ``` If no manifest, use a different tag like `node:20-alpine`. Rollback: revert the Dockerfile `FROM` line to the previous base image and rebuild. ### Dependency Installation Failure in Build Stage A package install step fails due to version conflicts or missing native libraries. Example with Node: ``` > node-gyp rebuild make: g++: No such file or directory ``` Fix: add build tools to the build stage only: ```dockerfile FROM node:18 AS builder RUN apt-get update && apt-get install -y python3 make g++ ``` These tools are not included in the final stage, so image size remains small. ### Runtime Crash After Successful Build The container builds but exits immediately with a non-zero code. Check logs: ```bash docker logs upgrade-test ``` Common error: missing shared library or environment variable. For example: ``` Error: Cannot find module 'express' ``` This means the production dependencies were not installed. In the Dockerfile, ensure `npm ci --only=production` runs in the final stage or copy `node_modules` from builder. Rollback: stop and remove the broken container, then run the old image: ```bash docker stop upgrade-test && docker rm upgrade-test docker run -d --name prod-old -p 3000:3000 myapp:old-prod ``` ### Data Loss Due to Volume Misconfiguration If the new container cannot find existing data, check volume mounts. ```bash docker inspect upgrade-test | jq '.[0].Mounts' ``` If the expected volume is missing, recreate the container with the correct `-v` flag: ```bash docker run -d --name upgrade-test -v app_data:/var/lib/app -p 3000:3000 myapp:upgrade-test ``` Never delete named volumes until you have verified the new container works. ## Operations Checklist Use this checklist before and during a multi-stage build upgrade. Each item has an owner and a review frequency. | Step | Responsible | Action | Verify | Revisit | |------|-------------|--------|--------|---------| | 1. Document current versions | DevOps Engineer (e.g., Priya Shah) | Run `docker version` and `docker buildx version`, save output | Versions recorded in runbook | Annually or before any upgrade | | 2. Review Dockerfile for multi-stage patterns | Application Developer (e.g., Marcus Lee) | Identify all `FROM`, `AS`, `COPY --from` lines | No missing stage names | Every Dockerfile change | | 3. Choose target Docker/BuildKit version | DevOps Engineer | Check release notes and compatibility matrix | Target version noted in ticket | Per upgrade cycle | | 4. Build test image with new version | CI Pipeline | Run `docker build -t myapp:upgrade-test .` in staging | Build completes without errors | Every build | | 5. Run smoke tests on test container | QA Engineer (e.g., Sofia Garcia) | Execute health check and functional tests | All tests pass | Every release candidate | | 6. Compare image sizes and layers | DevOps Engineer | `docker images` and `docker history` | New image size <= old image size or justified | Every build | | 7. Rollback plan documented | DevOps Engineer | Write rollback steps in runbook | Runbook updated and reviewed | Before production deployment | | 8. Monitor production after deployment | Site Reliability Engineer (e.g., Tom Okafor) | Watch logs, metrics, and error rates for 24 hours | No new error spikes | Immediately after deploy, then weekly for first month | Accountability: each item has a single named owner. Review frequency is specified to ensure ongoing accuracy. ## Common Pitfalls and How to Avoid Them ### Not Pinning Base Image Versions Using `FROM node:latest` or `FROM ubuntu` leads to unexpected changes when the base image updates. Always pin the major and minor version: ```dockerfile FROM node:18.17.1 AS builder ``` Or use digest: ```dockerfile FROM node:18@sha256:... AS builder ``` This makes builds reproducible. ### Copying Unnecessary Files into Final Stage A common mistake is `COPY --from=builder /app /app`, which includes source code, build tools, and secrets. Instead, copy only the runtime artifacts: ```dockerfile COPY --from=builder /app/dist ./dist COPY --from=builder /app/package.json ./ RUN npm ci --only=production ``` This reduces image size and attack surface. ### Ignoring Build Cache Invalidation Multi-stage builds can cache layers incorrectly if the order of COPY and RUN is not optimal. For Node.js, copy `package.json` and `package-lock.json` first, then run `npm ci`, then copy the rest of the source. This way, dependency installation is cached unless those files change. Before: ```dockerfile COPY . . RUN npm ci ``` After: ```dockerfile COPY package*.json ./ RUN npm ci COPY . . ``` Now, changes to source files do not invalidate the dependency cache. ### Skipping Rollback Testing Many teams test the upgrade but not the rollback. Always test rolling back to the previous image version. ```bash docker run -d --name rollback-test -p 3000:3000 myapp:old-prod curl http://localhost:3000/health ``` If the rollback image works, you have a safe path. ### Not Using BuildKit Secrets If your build stage needs access to private packages or credentials, avoid passing secrets as build args because they persist in image history. Use BuildKit secrets: ```dockerfile RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci ``` Build with: ```bash docker build --secret id=npmrc,src=$HOME/.npmrc -t myapp:secure . ``` This keeps secrets out of the final image. ## Conclusion Upgrading and migrating Docker multi-stage builds is a controlled process, not a single command. By documenting your current state, making scoped changes, verifying each step, and planning rollback, you protect your deployment pipeline and application runtime. The key takeaways: - Always check Docker and BuildKit versions before changing anything - Update the Dockerfile incrementally, stage by stage - Test the new image in isolation with a different tag - Compare image sizes and contents to ensure a leaner final image - Have a rollback plan and test it - Assign accountability and review checklists regularly Use this guide as a template for your own upgrade. Adapt the commands and examples to your specific application and environment, and you will reduce risk and downtime. Next step: pick one small multi-stage build from your project, run the inventory commands, and create a baseline. Then, attempt a version upgrade in a test environment and verify the results. Document what worked and what did not, and you will be better prepared for larger migrations.