Docker Registry Troubleshooting with Practical Examples

Introduction

Docker Registry is the core service that stores and distributes container images, so when it fails, teams cannot deploy applications. A single misconfigured storage backend, an expired TLS certificate, or a full disk can bring entire pipelines to a halt. Troubleshooting a registry effectively means working through the problem in layers: first inventory the environment, then make safe changes, verify behavior with concrete commands, and know how to recover if something goes wrong.

This guide is built for practitioners, whether you are a platform engineer managing a production registry or a developer fixing a local one. Every major step includes a real command or configuration snippet with explanatory output so you can follow along. By the end, you will have a repeatable checklist, common failure modes with recovery steps, and preventive practices to keep your registry healthy.

Version and Environment Inventory

Before changing anything, collect hard facts about your deployment. Knowing the registry version, storage backend, authentication method, network layout, and client versions prevents guesswork. Keep a simple inventory table; update it whenever the environment changes. For each item below, we provide a command you can run now.

Identify Registry Version

The official Docker Registry image is versioned by tag. If your registry runs in a container, inspect the image tag:

docker inspect <registry-container> --format '{{.Config.Image}}'

Expected output example:

registry:2.8.3

If the registry was installed from a package, query the package manager:

rpm -q docker-registry      # RHEL/CentOS
dpkg -l docker-registry     # Debian/Ubuntu

A healthy registry also answers the API version endpoint:

curl -v http://localhost:5000/v2/

Look for a 200 OK response and an empty JSON body {}. If you get a connection refused or timeout, the service is not running or is listening on a different port.

Check Storage Backend

The storage backend is where image layers and manifests live. Common backends include local filesystem, Amazon S3, Azure Blob Storage, Google Cloud Storage, and OpenStack Swift. Misconfigured credentials, missing buckets, or wrong permissions are frequent causes of 500 errors during push and pull.

Inspect the registry configuration file. For a containerized registry, print it directly:

docker exec <registry-container> cat /etc/docker/registry/config.yml

Look for the storage section. Here is a filesystem backend example:

storage:
  filesystem:
    rootdirectory: /var/lib/registry

And an S3 backend example:

storage:
  s3:
    accesskey: AKIAIOSFODNN7EXAMPLE
    secretkey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    region: us-east-1
    bucket: my-registry-bucket

Note the exact storage type and verify that credentials are valid from the registry host. For S3, test with:

aws s3 ls s3://my-registry-bucket --region us-east-1

For filesystem, check available disk space:

df -h /var/lib/registry

Authentication and TLS Configuration

If your registry is private, it likely uses basic authentication, token authentication, or TLS client certificates. Inspect the auth and http sections of the same config file. Example with TLS and basic auth:

http:
  addr: 0.0.0.0:5000
  tls:
    certificate: /certs/domain.crt
    key: /certs/domain.key
auth:
  htpasswd:
    realm: basic-realm
    path: /auth/htpasswd

For basic auth, verify the htpasswd file exists and is readable by the registry process. For TLS, check the certificate expiration date:

openssl x509 -enddate -noout -in /certs/domain.crt

Expected output:

notAfter=Mar 14 12:00:00 2026 GMT

If the certificate is expired or the htpasswd file path is wrong, clients will receive 401 or certificate verification errors.

Network Topology and Client Versions

Map how clients reach the registry. Is it directly on port 5000? Behind a reverse proxy like nginx or Traefik? Behind a load balancer? Each layer can introduce timeouts, header rewrites, or TLS termination issues.

Check the Docker client version on a machine that pushes or pulls:

docker version --format '{{.Client.Version}}'

Example:

24.0.5

Docker Registry API v2 is backward compatible, but very old clients may not support newer manifest formats. If you suspect a client issue, test with a known-good client.

Record all findings in a table like this:

Safe Configuration Path

Changing registry configuration in production without a plan can extend downtime and cause data loss. Treat configuration like code: test in staging, back up first, and apply one change at a time with rollback ready.

Backup Before Changes

Always back up the registry configuration file and the storage data if feasible. For a containerized registry, copy the config:

docker cp <registry-container>:/etc/docker/registry/config.yml config.yml.backup

For local filesystem storage, create a tarball:

tar -czf registry-data-backup-$(date +%Y%m%d).tar.gz /var/lib/registry

Store backups outside the registry host, such as in object storage or a separate server.

Test in Staging

If you have resources, run a throwaway registry locally to test changes before applying them to production. Start a test instance on port 5001 with a temporary volume:

docker run -d -p 5001:5000 --name registry-test \
  -v /tmp/registry-test:/var/lib/registry \
  registry:2.8.3

Push and pull a test image to confirm basic functionality. Only after the test passes, apply the same change to production.

Gradual Configuration Changes

When editing the main config file, change one parameter at a time. For example, if you need to switch the storage backend from filesystem to S3, first create a staging config with the S3 settings and test it thoroughly. Use version control for your config files; a Git repository makes rollback trivial.

After editing, restart the registry. With a plain Docker container, remove and recreate it with the updated mount:

docker stop registry && docker rm registry
docker run -d --name registry -p 5000:5000 \
  -v /path/to/config.yml:/etc/docker/registry/config.yml \
  -v registry-data:/var/lib/registry \
  registry:2.8.3

With Docker Compose or systemd, use the normal restart command:

docker compose restart registry
# or
systemctl restart docker-registry

Verification and Diagnostics

Once the registry is running, verify its health and investigate issues with logs, API calls, and functional tests.

Check Registry Logs

Logs are the first place to look for any problem. For a containerized registry:

docker logs --tail 100 <registry-container>

For a systemd service:

journalctl -u docker-registry -n 100 -f

Common error patterns in logs:

• level=error msg="response completed with error" with HTTP status codes 500 or 503 indicates an internal error.
• error resolving backend or AccessDenied points to storage backend issues.
• TLS handshake error suggests certificate problems.
• manifest unknown or blob unknown indicates missing or corrupted data.

Example log snippet showing a filesystem permission issue:

time="2025-02-01T10:00:00Z" level=error msg="error serving image" err="open /var/lib/registry/docker/registry/v2/repositories/...: permission denied"

API Health Check

Use curl to check the registry API. The /v2/ endpoint is the simplest health check:

curl -v http://localhost:5000/v2/

A healthy response includes HTTP/1.1 200 OK and an empty JSON body {}. If you receive 401 Unauthorized, authentication is required; provide credentials:

curl -u username:password -v http://localhost:5000/v2/

For a registry using a self-signed certificate, bypass verification with -k:

curl -k https://localhost:5000/v2/

If you receive 404 Not Found, the registry might be listening on a different path or the API version is not supported. Confirm the address and port.

Test Image Push and Pull

The ultimate functional test is pushing and pulling an actual image. Use a tiny image like alpine:

docker tag alpine:latest localhost:5000/alpine:test
docker push localhost:5000/alpine:test

Expected output shows progress bars and a final digest:

The push refers to repository [localhost:5000/alpine]
...
test: digest: sha256:abcdef... size: 528

Then test pulling from a clean state. Remove the local copy and pull again:

docker rmi localhost:5000/alpine:test
docker pull localhost:5000/alpine:test

If pull fails, check network connectivity, authentication, and whether the blob store is accessible.

Validate Storage Integrity

For filesystem storage, verify that repository directories exist and have correct permissions. The registry container typically runs as user root, but some deployments run as non-root. List the directory structure:

ls -l /var/lib/registry/docker/registry/v2/repositories

Expected output shows directories for each repository, e.g. alpine, busybox. If permissions are wrong, the registry logs will show permission denied errors. Correct with:

chown -R registry:registry /var/lib/registry

For S3 or other object storage, use their CLI tools to list objects and check access. For AWS S3:

aws s3 ls s3://my-registry-bucket/docker/registry/v2/repositories/

Failure Modes and Recovery

Knowing typical failure patterns helps you react quickly. For each failure mode below, we give symptoms, diagnosis steps, recovery actions, and rollback if the recovery fails.

Storage Backend Failures

Symptoms: 500 errors on push/pull, logs showing storage errors like NoSuchBucket, AccessDenied, connection refused, or timeout.

Diagnosis:

• Check storage configuration in the config file.
• For filesystem, check disk space: df -h /var/lib/registry.
• For S3, test bucket access: aws s3 ls s3://bucket-name --region us-east-1.
• Look at registry logs for specific error messages.

Recovery:

• If disk is full, free space by deleting unnecessary images or expanding the volume.
• If S3 permissions are wrong, update IAM policy or credentials, then restart the registry.
• If the bucket does not exist, create it in the correct region and ensure the registry user has write access.

Rollback: Restore the previous configuration backup and restart the registry. If data was written to the wrong backend, you may need to re-push or restore from backup.

Authentication and TLS Issues

Symptoms: 401 Unauthorized on all requests, certificate verification errors, client fails to connect with TLS handshake failure.

Diagnosis:

• Check the auth section in config; for basic auth, verify the htpasswd file exists and is readable by the registry process.
• Check certificate expiration: openssl x509 -enddate -noout -in domain.crt.
• Test with curl and -k to see if the issue is certificate-specific.

Recovery:

• Renew the TLS certificate and update the config file paths, then restart the registry.
• If basic auth credentials are wrong, regenerate the htpasswd file: htpasswd -Bbn user password > /auth/htpasswd.
• If using token auth, verify the token service endpoint is reachable and returning valid tokens.

Rollback: Restore the old certificate and original config, then restart.

Network and Proxy Problems

Symptoms: Timeouts, connection refused, registry unreachable from some clients but not others.

Diagnosis:

• Test connectivity from client to registry: nc -zv registry-host 5000.
• If behind a reverse proxy, check proxy logs for errors.
• Verify proxy headers: the registry expects X-Forwarded-Proto to be set correctly for TLS termination.

Recovery:

• Fix firewall rules to allow traffic on the registry port.
• Update proxy configuration to pass the correct headers, e.g. in nginx:

location /v2/ {
  proxy_pass http://registry:5000;
  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
}

• Restart the proxy service.

Rollback: Revert proxy config to the last known good version.

Data Corruption or Loss

Symptoms: Missing layers, manifest unknown or blob unknown errors, 500 errors on pull.

Diagnosis:

• Inspect logs for unknown blob or manifest errors.
• Run registry garbage collection in dry-run mode to see what would be removed:

docker exec <registry-container> bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml

• For filesystem storage, check for missing files or zero-byte files in repository directories.

Recovery:

• If corruption is limited, garbage collection might remove unreferenced blobs and fix inconsistencies. Run without dry-run only after backup.
• If severe, restore from the latest backup of the storage directory or volume.
• For object storage, check if bucket versioning or snapshots are available.

Rollback: Restore from backup and then re-push any images that were added after the backup.

Rollback Steps

Having a tested rollback plan is essential. A simple manual rollback looks like this:

# Stop the registry
docker stop <registry-container>

# Replace configuration with backup
cp config.yml.backup /etc/docker/registry/config.yml

# If needed, restore data
tar -xzf registry-data-backup.tar.gz -C /

# Restart the registry
docker start <registry-container>

# Verify
curl -v http://localhost:5000/v2/
docker push localhost:5000/alpine:test

If the rollback fails, do not leave the registry in an unknown state. Go back to the last known good configuration and data, even if it means temporarily losing recent changes. Avoid making multiple changes at once; this reduces the chance of an unrecoverable state.

Common Pitfalls and How to Avoid Them

Even experienced operators fall into these traps. Recognizing them early saves hours of debugging.

1. Editing Configuration Without a Backup

Why it happens: Under pressure, someone edits config.yml directly and restarts the registry, then discovers the new setting breaks everything.

How to avoid: Always copy the config to a backup file before editing. Better, keep the config in version control and only deploy tested changes from the repository.

How to recover: Restore the backup immediately, restart, and verify.

2. Ignoring Disk Space Until It Fills Up

Why it happens: Registries grow silently, and a full disk causes push failures and sometimes data corruption.

How to avoid: Set up monitoring with alerts at 70% and 85% disk usage. Schedule regular garbage collection with the registry's built-in tool.

How to recover: Free space by deleting unused images or expanding the volume. Then run garbage collection to clean up orphaned blobs.

3. Not Testing Client Connectivity After Network Changes

Why it happens: A firewall or security group change breaks connectivity, but the operator only looks at server-side logs.

How to avoid: Always test from a client machine with nc -zv registry-host 5000 and a real docker pull before closing the incident.

How to recover: Roll back the network change or open the necessary port and protocol.

4. Forgetting to Set Forwarded Headers Behind a Proxy

Why it happens: TLS termination at a proxy without setting X-Forwarded-Proto leads to incorrect URLs in the registry response, breaking pulls.

How to avoid: In proxy config, always set X-Forwarded-Proto and X-Forwarded-For. Test with curl and inspect response headers.

How to recover: Add the missing header, restart the proxy, and verify with a pull.

5. Running Garbage Collection Without a Dry Run

Why it happens: The operator runs garbage-collect in an attempt to free space, but it deletes blobs that are still referenced due to a misconfiguration, causing manifest failures.

How to avoid: Always run with --dry-run first and inspect the output. Perform a full backup before a real run.

How to recover: Restore from backup or re-push missing images.

Operations Checklist

Use this checklist during routine operations and troubleshooting. For each major item, we name a single accountable owner to avoid ambiguity when something goes wrong. Review the checklist monthly, or immediately after any incident.

Conclusion

Docker Registry troubleshooting is a skill that combines methodical investigation with practical command-line fluency. By following the layered approach in this guide, from environment inventory to safe recovery, you can resolve most issues without escalating to panic. Start with a narrow, measurable problem: for example, "push fails with 500 error," then apply the diagnostic steps in order. Use the checklist to keep your registry healthy and to share knowledge across the team. Document your fixes in a runbook, and practice recovery in a lab so that when a real incident occurs, the response is almost automatic. Remember, the goal is not just to fix the registry, but to prevent the same failure from happening twice.