## Intro Docker images are the building blocks of containerized applications, but they can grow in size and number without warning. Without a capacity plan, you may find your registry storage full, deployment times slowing, and developers unable to push new images. This guide provides a practical approach to planning Docker image capacity, including estimating storage, setting limits, monitoring growth, and planning for safe operations. We will use concrete commands and examples that you can adapt to your environment. ## Version and Environment Inventory Before planning capacity, you need to know your current Docker environment. This includes the Docker version, the storage driver, the registry type, and the images currently in use. ### Docker Version and Storage Driver Run the following commands to get Docker version and storage driver: ```bash docker version --format '{{.Server.Version}}' docker info --format 'Storage Driver: {{.Driver}}' ``` Example output: ``` 20.10.12 Storage Driver: overlay2 ``` The storage driver affects how images are stored and deduplicated. overlay2 is common and efficient for most Linux distributions. If you use a different driver, note its characteristics. ### Registry and Image Inventory List all images on a host: ```bash docker images ``` Sample output: ``` REPOSITORY TAG IMAGE ID CREATED SIZE myapp latest abc123def456 2 hours ago 450MB myapp v1.0 def456abc123 1 week ago 445MB nginx latest nginx123abc 2 weeks ago 142MB ``` To get a summary of total disk usage by Docker images: ```bash docker system df ``` Output: ``` TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 12 3 4.2GB 2.1GB (50%) Containers 5 2 200MB 150MB (75%) Local Volumes 2 1 500MB 250MB (50%) Build Cache 0 0 0B 0B ``` This shows the total size of images and how much is reclaimable. In this example, you could free 2.1GB by removing unused images. ### Prerequisites - Docker installed (version 20.x or later for most features). - Access to a Docker registry if using a remote one (e.g., Docker Hub, AWS ECR, or a private registry). - Sufficient disk space to hold images and overhead (typically 20-30% extra for build cache and metadata). - Understanding of your application's image lifecycle. ## Safe Configuration Path Now, let's implement capacity planning measures. The key is to start with a narrow, measurable pilot: choose one application or a small set of images, and apply the following techniques. ### 1. Optimize Dockerfile to Reduce Image Size Smaller images consume less storage and transfer faster. Use multi-stage builds, minimal base images, and remove unnecessary files. Example original Dockerfile: ```dockerfile FROM node:14 WORKDIR /app COPY package*.json ./ RUN npm install COPY . . RUN npm run build CMD ["npm", "start"] ``` Optimized version using multi-stage and alpine: ```dockerfile # Build stage FROM node:14-alpine AS build WORKDIR /app COPY package*.json ./ RUN npm install COPY . . RUN npm run build # Production stage FROM node:14-alpine WORKDIR /app COPY --from=build /app/dist ./dist COPY package*.json ./ RUN npm install --only=production CMD ["npm", "start"] ``` Build the optimized image and compare sizes: ```bash docker build -t myapp:optimized . docker images myapp ``` Output before and after: ``` myapp latest 450MB myapp optimized 120MB ``` A reduction of over 70%. ### 2. Set Image Retention Policies Automatically remove old or untagged images to prevent accumulation. Use `docker image prune` with filters. To remove dangling images: ```bash docker image prune ``` To remove all images not used by any container (including tagged): ```bash docker image prune -a ``` Output example: ``` Deleted Images: untagged: myapp@sha256:... deleted: sha256:... untagged: myapp:v1.0 ... Total reclaimed space: 2.1GB ``` Schedule this as a cron job or use registry-level policies (e.g., AWS ECR lifecycle policies) for remote registries. Example ECR lifecycle policy JSON: ```json { "rules": [ { "rulePriority": 1, "description": "Keep last 10 images", "selection": { "tagStatus": "any", "countType": "imageCountMoreThan", "countNumber": 10 }, "action": { "type": "expire" } } ] } ``` This would expire all but the 10 most recent images in the repository. ### 3. Monitor and Limit Registry Storage If you run a private registry, set storage quotas. For example, with Docker Registry 2.0, you can use configuration to limit storage or rely on underlying filesystem quotas. If using a cloud provider, set up alerts in CloudWatch or Stackdriver when registry storage approaches a threshold. Example: AWS CLI to check ECR repository size (using `aws ecr describe-images` is complex; here is a simplified approach using a script). ```bash aws ecr describe-repositories --query 'repositories[].[repositoryName, repositoryUri]' ``` To get the size of a repository, you may need to use a custom script that sums image sizes from `describe-images`. ### 4. Use Image Tagging Strategically Adopt a tagging strategy that supports capacity management. For example, use immutable tags for releases (e.g., `v1.2.3`) and avoid using `latest` for production. This prevents unexpected overwrites and allows better tracking. Example: ```bash docker tag myapp:latest myapp:v1.2.3 docker push myapp:v1.2.3 ``` Then, you can safely prune old tags after a certain period. ## Verification and Diagnostics After implementing these measures, verify that they are working as expected. ### Check Image Sizes Regularly monitor image sizes. Use a script to list images and sort by size: ```bash docker images --format '{{.Repository}}:{{.Tag}} {{.Size}}' | sort -rh -k2 ``` Output: ``` myapp:latest 450MB myapp:v1.0 445MB nginx:latest 142MB ... ``` This helps identify large images that may need optimization. ### Check Reclaimable Space Run `docker system df` to see reclaimable space: ```bash docker system df ``` If you have run a prune, the reclaimable should be lower. ### Test Image Pull Time Measure how long it takes to pull an image on a fresh node, which indicates transfer time and can affect scaling. ```bash time docker pull myapp:latest ``` Output: ``` real 0m12.345s user 0m0.123s sys 0m0.456s ``` If this is too long, consider a local registry or a content distribution network. ### Monitor Registry API For a private registry, check the API for storage usage (if available). For Docker Hub, you can use the API to get repository details, but it may have rate limits. Example curl to Docker Hub API: ```bash curl -s https://hub.docker.com/v2/repositories/library/nginx/ | jq .pull_count ``` Output shows pull count, which may indicate popularity but not size. ### Validate Retention Policy After setting up a lifecycle policy (e.g., in ECR), validate that images are expired as expected. You can check by listing images after the policy evaluation period. ## Failure Modes and Recovery Capacity planning failures can lead to several issues. Here are common failure modes and how to recover. ### Disk Full on Docker Host Symptom: Docker commands fail with "no space left on device" or pulls are stuck. Recovery: - Identify large files: `du -sh /var/lib/docker` - Remove unused images: `docker image prune -a` - If necessary, remove stopped containers and volumes: `docker system prune` - Expand disk or move Docker data directory to a larger disk. Example moving Docker data directory (requires stop docker): ```bash sudo systemctl stop docker sudo rsync -aP /var/lib/docker/ /mnt/large/docker/ sudo rm -rf /var/lib/docker sudo ln -s /mnt/large/docker /var/lib/docker sudo systemctl start docker ``` ### Registry Out of Storage Symptom: Push fails with "denied: requested access to the resource is denied" or "out of space". Recovery: - For self-hosted registry, expand storage or delete unreferenced layers. - For cloud registry, increase storage quota (may incur costs) or clean up old images using lifecycle policies. ### Slow Image Pulls Symptom: Deployments take longer, and new nodes take minutes to start. Recovery: - Optimize image size as shown earlier. - Use a local registry mirror or a caching proxy. - For Kubernetes, use image pre-pulling on nodes. ### Incorrect Tagging Leads to Overwrites Symptom: Production suddenly runs different code because `latest` was overwritten. Recovery: - Always use immutable tags for production. - Roll back by deploying the previous tagged image. - Implement a policy where only CI/CD can push to production tags. ### Rollback Plan Always have a rollback plan for any configuration changes. For example, if you implemented a new retention policy and it deletes too many images, you may need to revert the policy and restore images from backups (if available). Keep a backup of critical images by pushing to a separate repository or archiving to a storage service. ## Common Pitfalls and Mistakes Even with careful planning, teams often encounter these issues. Understanding why they happen and how to avoid them can save time and storage. ### Ignoring Build Cache Growth Why it happens: Docker build cache accumulates on build servers, especially when builds are frequent and Dockerfiles change. How to avoid: Regularly prune build cache using `docker builder prune` on a schedule. Set a build cache size limit in Docker daemon configuration if your environment supports it. ### Overusing the `latest` Tag Why it happens: Developers default to `latest` for convenience, leading to unpredictable image content and difficulty rolling back. How to avoid: Enforce immutable tags for releases. Use a tagging convention like `major.minor.patch` or `commit-sha`. Block pushes to `latest` in production registries using policy. ### Neglecting Multi-Stage Builds Why it happens: Teams stick with single-stage builds because they work, not realizing the size impact. How to avoid: Adopt multi-stage builds for all new projects. Review existing Dockerfiles and refactor where possible. Show the size difference to stakeholders to justify the effort. ### Underestimating Image Proliferation Why it happens: Each feature branch or test build creates new images without a cleanup strategy. How to avoid: Implement registry lifecycle policies from day one. Use automatic tag cleanup for temporary builds. Monitor image counts and set alerts when thresholds are crossed. ### Ignoring Base Image Updates Why it happens: Base images are rarely updated, leading to outdated and potentially larger images. How to avoid: Regularly rebuild images with updated base images. Use tools like Dependabot or Renovate to automate base image updates in Dockerfiles. ## Operations Checklist Use this checklist for ongoing capacity management. Assign each item to a single accountable owner and set a review frequency. | Item | Owner | Frequency | |------|-------|-----------| | Monitor image count and total size using `docker system df` | DevOps Engineer | Weekly | | Review registry storage usage and set alerts at 70% and 90% thresholds | Platform Lead | Monthly | | Run `docker image prune -a` on build servers | CI/CD Admin | Daily | | Apply lifecycle policies on remote registries to expire old images | DevOps Engineer | Weekly (check) | | Review and optimize Dockerfiles for any new images; enforce size limits in CI/CD | Application Architect | Monthly | | Test image pull times in staging before production | QA Lead | Per Release | | Maintain a tag naming convention and enforce immutability for releases | Engineering Manager | Quarterly Review | | Document image retention policies and ensure team awareness | DevOps Engineer | Quarterly | | Perform periodic capacity reviews and adjust limits based on growth trends | Platform Lead | Monthly | | Have a runbook for disk full and registry full incidents | SRE | Quarterly (test) | ## Conclusion Docker image capacity planning is essential for maintaining a healthy container environment. By inventorying your current images, optimizing Dockerfiles, implementing retention policies, and monitoring usage, you can prevent storage exhaustion and deployment bottlenecks. Start with a small, measurable pilot: choose one application, apply the techniques in this guide, and verify the results. Then expand to your entire fleet. Use the operations checklist to maintain control and adjust as your needs grow. With these practices, you can ensure that your Docker images remain efficient, manageable, and cost-effective. ### Summary of Key Commands | Command | Purpose | |---------|---------| | `docker system df` | Show disk usage by Docker components | | `docker images` | List images and sizes | | `docker image prune -a` | Remove all unused images | | `docker build -t myapp:optimized .` | Build optimized image | | `docker tag source target` | Apply new tag | | `time docker pull image` | Measure pull time | By following this guide, you can establish a solid capacity planning strategy for Docker images.