Docker Registry Common Errors and Fixes: A Practical Operations Guide

Intro

A private Docker Registry is a critical piece of infrastructure for many development teams. It stores and serves container images, and when it fails, it can block CI/CD pipelines, prevent deployments, and frustrate developers. This guide focuses on common Docker Registry errors and how to fix them with practical examples. We cover configuration mistakes, storage issues, authentication problems, and operational pitfalls. Each section provides concrete commands, expected outputs, and recovery steps.

This article is written for operators, DevOps engineers, and developers who manage a self-hosted Docker Registry (the registry:2 image from Docker, often called Distribution). It assumes you have basic Docker and Linux command-line skills and access to the registry host. We emphasize a safe approach: observe before changing, limit the blast radius, always use placeholders instead of real secrets in examples, and verify after every fix. We will not cover managed registries like Docker Hub, Amazon ECR, or Google Artifact Registry, except occasionally for comparison. The goal is to help you quickly diagnose and resolve issues with your own registry.

Version and Environment Inventory

Before troubleshooting, you need a clear picture of your registry setup. Knowing the exact version, deployment method, and storage backend is essential because errors and fixes differ across versions.

Start by finding the registry container and its version. Run:

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"

Look for a container using an image like registry:2 or a custom image. To get the exact version, execute inside the container:

docker exec <registry-container> registry --version

Expected output example:

registry github.com/docker/distribution v2.8.3+unknown

If you are running the registry under Docker Compose or Kubernetes, find the manifest or deployment file. For Compose, docker compose ps and docker compose config show the current state and effective configuration. For Kubernetes, use kubectl get pods -l app=registry -o yaml and kubectl describe pod <pod-name>.

Also, note how the registry is configured. The main configuration file is typically /etc/docker/registry/config.yml inside the container, but it may be mounted from the host. Inspect the container mounts:

docker inspect <registry-container> --format '{{ json .Mounts }}' | jq

This shows volumes and bind mounts, which are crucial for storage troubleshooting. For example:

[
  {
    "Type": "bind",
    "Source": "/opt/registry/data",
    "Destination": "/var/lib/registry",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  }
]

In this case, data is stored on the host at /opt/registry/data. If this path is not persistent or has incorrect permissions, the registry will fail to write images.

Always capture the current state before making changes. Save the output of docker inspect, the config file, and logs to a temporary directory. This gives you a rollback point and helps in post-mortem analysis. For example:

mkdir -p /tmp/registry-debug-$(date +%Y%m%d-%H%M%S)
docker inspect <registry-container> > /tmp/registry-debug-*/inspect.json
docker logs <registry-container> --tail 200 > /tmp/registry-debug-*/logs.txt

Use placeholders for secrets in any command you share or document. Do not paste real passwords or tokens.

Safe Configuration Path

Many registry errors stem from misconfiguration. The registry uses a YAML file, commonly config.yml. Let's examine a minimal configuration and then discuss common mistakes.

A basic configuration that uses the local filesystem for storage and allows anonymous pulls might look like this:

version: 0.1
log:
  level: info
  fields:
    service: registry
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: :5000

If you need authentication, you'd add an auth section, often using htpasswd:

auth:
  htpasswd:
    realm: basic-realm
    path: /auth/htpasswd

But a common error is mismatched paths between the config file and the volume mounts. For instance, if you change rootdirectory to /var/lib/registry but the volume is mounted at /var/lib/registry, ensure the mount is correct. If you use a different path, like /data, but the mount is at /var/lib/registry, the registry writes to the container layer, and data disappears on container recreation.

To safely change configuration:

1. Copy the current config file from the container or host.

docker cp <registry-container>:/etc/docker/registry/config.yml /tmp/config.yml.backup

1. Make your changes in a test environment. If using Docker Compose, you can create a separate compose file for testing with a different port and storage.
2. Validate the YAML syntax. Use docker run --rm -v /tmp/config.yml:/config.yml -it registry:2 registry serve /config.yml but it will start serving; instead, you can use a YAML linter like yamllint on the host. Inside the container, you can use python -c 'import yaml, sys; yaml.safe_load(sys.stdin)' < config.yml if Python is available.
3. Apply the change by recreating the container with the new config (mount the file read-only).
4. Monitor logs: docker logs -f <registry-container>.

For configuration changes, always keep the blast radius small. Change one setting at a time, and document the expected outcome. If the registry fails to start, revert to the backup config and investigate.

A practical example: enabling the delete API to allow removing images. Many users hit "Method Not Allowed" errors when trying to delete an image because the delete feature is not enabled by default in the config. The error message from a client like docker push or curl might be:

HTTP 405 Method Not Allowed

The fix is to set delete.enabled: true under storage as shown above. After changing, restart the registry and test with a curl -X DELETE request to an image manifest. Note that enabling delete does not immediately free disk space; you need to run garbage collection (see Storage section).

Verification and Diagnostics

After any change, you must verify that the registry behaves as expected. Start with health checks. The registry image includes a healthcheck if configured, but you can also use curl to the /v2/ endpoint.

Run from the host or inside the container:

curl -v http://localhost:5000/v2/

Expected successful response:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0

{}

If you get a 404 or connection refused, the registry is not listening properly or the port mapping is wrong. Check docker port <registry-container> and docker logs for errors like listen tcp :5000: bind: address already in use.

Another common diagnostic is checking image push and pull. Use a small test image, tag it to your local registry, and push:

docker pull alpine:latest
docker tag alpine:latest localhost:5000/alpine:test
docker push localhost:5000/alpine:test

Watch the output for errors. A successful push ends with a digest line. If authentication is enabled, you need to docker login localhost:5000 first.

For deeper diagnostics, inspect the registry's API with curl to list repositories and tags:

curl -s http://localhost:5000/v2/_catalog
curl -s http://localhost:5000/v2/alpine/tags/list

Expected output:

{"repositories":["alpine"]}
{"name":"alpine","tags":["test"]}

If you see errors, check the registry logs. Increase log level to debug in the config temporarily for more verbose output. Remember to revert to info after troubleshooting.

Failure Modes and Recovery

Here are specific failure modes with their fixes.

1. dial tcp: lookup registry on 127.0.0.53:53: no such host when pulling from a remote registry

This error occurs when using a registry as a pull-through cache or when the registry tries to contact a remote registry but DNS resolution fails. Check the registry's network configuration, especially DNS settings. If running in Docker, ensure the container has proper DNS. You can specify DNS in the Docker run command with --dns. Alternatively, if using a proxy cache, ensure the remoteurl in the config is correct and accessible.

2. manifest unknown error on pull

When you push an image and then someone else tries to pull it, they might get manifest unknown. This often happens when the registry is configured with multiple storage backends (like S3 and filesystem) and the manifest is stored in one but not the other, or when the same repository name is used with different casing or tags. Verify consistency: check docker tag and ensure the repository name matches exactly. Also check if the registry is behind a load balancer that is routing to different instances without shared storage.

3. blob upload unknown during push

This error suggests that the registry cannot find the upload session, possibly because the session timed out or the registry restarted mid-upload. Increase the http timeouts in the config or ensure the client retries. A typical fix is to set larger values for http section:

http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
  timeouts:
    read: 900
    write: 900
    idle: 900

Restart and retry the push.

4. denied: requested access to the resource is denied

This is an authentication or authorization error. Check:

• Whether you are logged in (docker login).
• If the credentials are correct (htpasswd file).
• If the user has permission if using a more complex auth like token-based.

Use curl -u username:password https://registry.example.com/v2/ to test. If the credentials are wrong, the server returns 401. Verify the htpasswd file is mounted correctly and the user exists: docker exec <registry-container> htpasswd -B -C 10 /auth/htpasswd username (note: htpasswd may not be in the image; you can use registry:2 with htpasswd installed separately or use a tool on the host).

5. Out of disk space on the registry host

This is a silent failure: pushes may hang or fail with io timeout. Check disk usage: df -h on the host. If the registry's storage directory is on a full partition, you must free space. Often, the culprit is old unreferenced blobs. Run garbage collection (see Storage section). Also consider setting up monitoring and alerts for disk usage.

Operations Checklist

Use this checklist to systematically troubleshoot and maintain your Docker Registry. It follows the observe-change-verify cycle.

Each item has a single owner (not a team) to ensure accountability. Revisit this checklist quarterly or when the registry version changes.

Storage and Data Integrity

Storage issues are among the most common and most dangerous. The registry stores images as blobs (layers) and manifests. If storage becomes corrupted or misconfigured, you may lose data.

Filesystem vs. Object Storage: The default is a local filesystem, but for production, object storage like S3, GCS, or Azure Blob is recommended for durability and scalability. Mixing storage types or changing storage backend without migrating data leads to manifest unknown.

Permissions: The registry process runs as user root in the container by default, but if you run it with a non-root user, ensure the storage directory is writable. A common error is open /var/lib/registry/docker/registry/v2/repositories: permission denied in the logs. Fix it by changing the ownership on the host directory: sudo chown -R 1000:1000 /opt/registry/data (assuming UID 1000 is the container user). Adjust the UID/GID as per your setup.

Garbage Collection: Over time, especially if you delete images or overwrite tags, unreferenced blobs accumulate. Run garbage collection periodically. The registry must be in read-only mode during collection to avoid corruption. Here is a safe procedure:

1. Stop the registry container or put it in read-only mode. You can do this by setting storage.maintenance.readonly.enabled: true in config and restarting, or simply stop the container.
2. Run garbage collection from the same image, mounting the same config and data:

docker run --rm -v /opt/registry/data:/var/lib/registry -v /opt/registry/config.yml:/etc/docker/registry/config.yml registry:2 bin/registry garbage-collect /etc/docker/registry/config.yml

Expected output lists the blobs marked for deletion.

1. Restart the registry normally.

Backup: Always back up the registry data before running garbage collection or making significant changes. The backup should include the storage directory and the config file. Test restores regularly.

Security and Access Control

A misconfigured registry can expose private images or allow unauthorized pushes. Here are key mistakes and fixes:

• Running without TLS: In production, always use HTTPS. A registry without TLS requires Docker clients to be configured with insecure-registries, which is a security risk. Use a reverse proxy like Nginx or Caddy to terminate TLS and forward to the registry. A basic Nginx config:

server {
    listen 443 ssl;
    server_name registry.example.com;

    ssl_certificate /etc/ssl/certs/registry.crt;
    ssl_certificate_key /etc/ssl/private/registry.key;

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

• Weak authentication: If using htpasswd, generate bcrypt hashes. Avoid SHA or MD5. Use htpasswd -B -C 12 for high cost. Never store plaintext passwords.
• Exposing the registry to the internet without auth: This allows anyone to push and pull. Always enable at least basic auth, and consider token-based auth for more granular control.
• Using default secrets in config: Avoid hardcoding tokens; use environment variables or secret mounts.

Common Pitfalls and How to Avoid Them

Here are frequent mistakes that lead to registry failures:

1. Not persisting storage - Using the default container filesystem and then losing all images when the container is recreated. Always mount a volume for /var/lib/registry and ensure it is backed up.
2. Misconfiguring the rootdirectory path - Setting a path in config that does not match the mount point, causing data to be written to ephemeral storage.
3. Ignoring log messages - The registry logs are detailed and often tell you the exact error. Set log level to info or debug and check logs before guessing.
4. Changing storage backend without migration - Switching from filesystem to S3 and expecting old images to be available. You must migrate data or start fresh.
5. Running garbage collection incorrectly - Doing it while the registry is writable or using a different configuration can corrupt data. Always follow the safe procedure.
6. Not monitoring disk and health - A full disk can silently break pushes. Set up monitoring with Prometheus and alert on high usage.
7. Overlooking security updates - The registry image may have vulnerabilities. Regularly update to the latest patch version.

Conclusion

Docker Registry errors can be frustrating, but with a systematic approach, they are manageable. Start by understanding your environment, make small, reversible changes, and verify after each step. Use the commands and examples in this guide to diagnose and fix common issues. Keep your configuration under version control, monitor the registry, and practice disaster recovery. By following these practices, you'll maintain a reliable and secure container image registry that supports your development workflow.

As a next step, pick one low-risk verification from this guide, such as testing the health endpoint or pushing a test image. Record the current state, run the check, and compare the result with the expected output. Then, review your storage and backup strategy to ensure your registry can survive failures.