## Introduction

Docker Registry is the core service that stores and distributes container images, so when it fails, teams cannot deploy applications. A single misconfigured storage backend, an expired TLS certificate, or a full disk can bring entire pipelines to a halt. Troubleshooting a registry effectively means working through the problem in layers: first inventory the environment, then make safe changes, verify behavior with concrete commands, and know how to recover if something goes wrong.

This guide is built for practitioners, whether you are a platform engineer managing a production registry or a developer fixing a local one. Every major step includes a real command or configuration snippet with explanatory output so you can follow along. By the end, you will have a repeatable checklist, common failure modes with recovery steps, and preventive practices to keep your registry healthy.

## Version and Environment Inventory

Before changing anything, collect hard facts about your deployment. Knowing the registry version, storage backend, authentication method, network layout, and client versions prevents guesswork. Keep a simple inventory table; update it whenever the environment changes. For each item below, we provide a command you can run now.

### Identify Registry Version

The official Docker Registry image is versioned by tag. If your registry runs in a container, inspect the image tag:

```bash
docker inspect <registry-container> --format '{{.Config.Image}}'
```

Expected output example:

```text
registry:2.8.3
```

If the registry was installed from a package, query the package manager:

```bash
rpm -q docker-registry      # RHEL/CentOS
dpkg -l docker-registry     # Debian/Ubuntu
```

A healthy registry also answers the API version endpoint:

```bash
curl -v http://localhost:5000/v2/
```

Look for a 200 OK response and an empty JSON body `{}`. If you get a connection refused or timeout, the service is not running or is listening on a different port.

### Check Storage Backend

The storage backend is where image layers and manifests live. Common backends include local filesystem, Amazon S3, Azure Blob Storage, Google Cloud Storage, and OpenStack Swift. Misconfigured credentials, missing buckets, or wrong permissions are frequent causes of 500 errors during push and pull.

Inspect the registry configuration file. For a containerized registry, print it directly:

```bash
docker exec <registry-container> cat /etc/docker/registry/config.yml
```

Look for the `storage` section. Here is a filesystem backend example:

```yaml
storage:
  filesystem:
    rootdirectory: /var/lib/registry
```

And an S3 backend example:

```yaml
storage:
  s3:
    accesskey: AKIAIOSFODNN7EXAMPLE
    secretkey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
    region: us-east-1
    bucket: my-registry-bucket
```

Note the exact storage type and verify that credentials are valid from the registry host. For S3, test with:

```bash
aws s3 ls s3://my-registry-bucket --region us-east-1
```

For filesystem, check available disk space:

```bash
df -h /var/lib/registry
```

### Authentication and TLS Configuration

If your registry is private, it likely uses basic authentication, token authentication, or TLS client certificates. Inspect the `auth` and `http` sections of the same config file. Example with TLS and basic auth:

```yaml
http:
  addr: 0.0.0.0:5000
  tls:
    certificate: /certs/domain.crt
    key: /certs/domain.key
auth:
  htpasswd:
    realm: basic-realm
    path: /auth/htpasswd
```

For basic auth, verify the htpasswd file exists and is readable by the registry process. For TLS, check the certificate expiration date:

```bash
openssl x509 -enddate -noout -in /certs/domain.crt
```

Expected output:

```text
notAfter=Mar 14 12:00:00 2026 GMT
```

If the certificate is expired or the htpasswd file path is wrong, clients will receive 401 or certificate verification errors.

### Network Topology and Client Versions

Map how clients reach the registry. Is it directly on port 5000? Behind a reverse proxy like nginx or Traefik? Behind a load balancer? Each layer can introduce timeouts, header rewrites, or TLS termination issues.

Check the Docker client version on a machine that pushes or pulls:

```bash
docker version --format '{{.Client.Version}}'
```

Example:

```text
24.0.5
```

Docker Registry API v2 is backward compatible, but very old clients may not support newer manifest formats. If you suspect a client issue, test with a known-good client.

Record all findings in a table like this:

| Component       | Version / Details                   | Notes                        |
|-----------------|-------------------------------------|------------------------------|
| Registry        | registry:2.8.3                      | Containerized                |
| Storage         | filesystem /var/lib/registry        | Local volume                 |
| TLS             | Self-signed certificate             | Valid until 2026             |
| Docker Client   | 24.0.5                              | On developer machine         |
| Reverse Proxy   | nginx 1.24                          | Terminates TLS               |
| Host OS         | Ubuntu 22.04                        | Kernel 5.15                  |

## Safe Configuration Path

Changing registry configuration in production without a plan can extend downtime and cause data loss. Treat configuration like code: test in staging, back up first, and apply one change at a time with rollback ready.

### Backup Before Changes

Always back up the registry configuration file and the storage data if feasible. For a containerized registry, copy the config:

```bash
docker cp <registry-container>:/etc/docker/registry/config.yml config.yml.backup
```

For local filesystem storage, create a tarball:

```bash
tar -czf registry-data-backup-$(date +%Y%m%d).tar.gz /var/lib/registry
```

Store backups outside the registry host, such as in object storage or a separate server.

### Test in Staging

If you have resources, run a throwaway registry locally to test changes before applying them to production. Start a test instance on port 5001 with a temporary volume:

```bash
docker run -d -p 5001:5000 --name registry-test \
  -v /tmp/registry-test:/var/lib/registry \
  registry:2.8.3
```

Push and pull a test image to confirm basic functionality. Only after the test passes, apply the same change to production.

### Gradual Configuration Changes

When editing the main config file, change one parameter at a time. For example, if you need to switch the storage backend from filesystem to S3, first create a staging config with the S3 settings and test it thoroughly. Use version control for your config files; a Git repository makes rollback trivial.

After editing, restart the registry. With a plain Docker container, remove and recreate it with the updated mount:

```bash
docker stop registry && docker rm registry
docker run -d --name registry -p 5000:5000 \
  -v /path/to/config.yml:/etc/docker/registry/config.yml \
  -v registry-data:/var/lib/registry \
  registry:2.8.3
```

With Docker Compose or systemd, use the normal restart command:

```bash
docker compose restart registry
# or
systemctl restart docker-registry
```

## Verification and Diagnostics

Once the registry is running, verify its health and investigate issues with logs, API calls, and functional tests.

### Check Registry Logs

Logs are the first place to look for any problem. For a containerized registry:

```bash
docker logs --tail 100 <registry-container>
```

For a systemd service:

```bash
journalctl -u docker-registry -n 100 -f
```

Common error patterns in logs:

- `level=error msg="response completed with error"` with HTTP status codes 500 or 503 indicates an internal error.
- `error resolving backend` or `AccessDenied` points to storage backend issues.
- `TLS handshake error` suggests certificate problems.
- `manifest unknown` or `blob unknown` indicates missing or corrupted data.

Example log snippet showing a filesystem permission issue:

```text
time="2025-02-01T10:00:00Z" level=error msg="error serving image" err="open /var/lib/registry/docker/registry/v2/repositories/...: permission denied"
```

### API Health Check

Use `curl` to check the registry API. The `/v2/` endpoint is the simplest health check:

```bash
curl -v http://localhost:5000/v2/
```

A healthy response includes `HTTP/1.1 200 OK` and an empty JSON body `{}`. If you receive `401 Unauthorized`, authentication is required; provide credentials:

```bash
curl -u username:password -v http://localhost:5000/v2/
```

For a registry using a self-signed certificate, bypass verification with `-k`:

```bash
curl -k https://localhost:5000/v2/
```

If you receive `404 Not Found`, the registry might be listening on a different path or the API version is not supported. Confirm the address and port.

### Test Image Push and Pull

The ultimate functional test is pushing and pulling an actual image. Use a tiny image like `alpine`:

```bash
docker tag alpine:latest localhost:5000/alpine:test
docker push localhost:5000/alpine:test
```

Expected output shows progress bars and a final digest:

```text
The push refers to repository [localhost:5000/alpine]
...
test: digest: sha256:abcdef... size: 528
```

Then test pulling from a clean state. Remove the local copy and pull again:

```bash
docker rmi localhost:5000/alpine:test
docker pull localhost:5000/alpine:test
```

If pull fails, check network connectivity, authentication, and whether the blob store is accessible.

### Validate Storage Integrity

For filesystem storage, verify that repository directories exist and have correct permissions. The registry container typically runs as user `root`, but some deployments run as non-root. List the directory structure:

```bash
ls -l /var/lib/registry/docker/registry/v2/repositories
```

Expected output shows directories for each repository, e.g. `alpine`, `busybox`. If permissions are wrong, the registry logs will show `permission denied` errors. Correct with:

```bash
chown -R registry:registry /var/lib/registry
```

For S3 or other object storage, use their CLI tools to list objects and check access. For AWS S3:

```bash
aws s3 ls s3://my-registry-bucket/docker/registry/v2/repositories/
```

## Failure Modes and Recovery

Knowing typical failure patterns helps you react quickly. For each failure mode below, we give symptoms, diagnosis steps, recovery actions, and rollback if the recovery fails.

### Storage Backend Failures

**Symptoms:** 500 errors on push/pull, logs showing storage errors like `NoSuchBucket`, `AccessDenied`, `connection refused`, or `timeout`.

**Diagnosis:**

- Check storage configuration in the config file.
- For filesystem, check disk space: `df -h /var/lib/registry`.
- For S3, test bucket access: `aws s3 ls s3://bucket-name --region us-east-1`.
- Look at registry logs for specific error messages.

**Recovery:**

- If disk is full, free space by deleting unnecessary images or expanding the volume.
- If S3 permissions are wrong, update IAM policy or credentials, then restart the registry.
- If the bucket does not exist, create it in the correct region and ensure the registry user has write access.

**Rollback:** Restore the previous configuration backup and restart the registry. If data was written to the wrong backend, you may need to re-push or restore from backup.

### Authentication and TLS Issues

**Symptoms:** 401 Unauthorized on all requests, certificate verification errors, client fails to connect with TLS handshake failure.

**Diagnosis:**

- Check the `auth` section in config; for basic auth, verify the htpasswd file exists and is readable by the registry process.
- Check certificate expiration: `openssl x509 -enddate -noout -in domain.crt`.
- Test with `curl` and `-k` to see if the issue is certificate-specific.

**Recovery:**

- Renew the TLS certificate and update the config file paths, then restart the registry.
- If basic auth credentials are wrong, regenerate the htpasswd file: `htpasswd -Bbn user password > /auth/htpasswd`.
- If using token auth, verify the token service endpoint is reachable and returning valid tokens.

**Rollback:** Restore the old certificate and original config, then restart.

### Network and Proxy Problems

**Symptoms:** Timeouts, connection refused, registry unreachable from some clients but not others.

**Diagnosis:**

- Test connectivity from client to registry: `nc -zv registry-host 5000`.
- If behind a reverse proxy, check proxy logs for errors.
- Verify proxy headers: the registry expects `X-Forwarded-Proto` to be set correctly for TLS termination.

**Recovery:**

- Fix firewall rules to allow traffic on the registry port.
- Update proxy configuration to pass the correct headers, e.g. in nginx:

```nginx
location /v2/ {
  proxy_pass http://registry:5000;
  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
}
```

- Restart the proxy service.

**Rollback:** Revert proxy config to the last known good version.

### Data Corruption or Loss

**Symptoms:** Missing layers, `manifest unknown` or `blob unknown` errors, 500 errors on pull.

**Diagnosis:**

- Inspect logs for unknown blob or manifest errors.
- Run registry garbage collection in dry-run mode to see what would be removed:

```bash
docker exec <registry-container> bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml
```

- For filesystem storage, check for missing files or zero-byte files in repository directories.

**Recovery:**

- If corruption is limited, garbage collection might remove unreferenced blobs and fix inconsistencies. Run without dry-run only after backup.
- If severe, restore from the latest backup of the storage directory or volume.
- For object storage, check if bucket versioning or snapshots are available.

**Rollback:** Restore from backup and then re-push any images that were added after the backup.

### Rollback Steps

Having a tested rollback plan is essential. A simple manual rollback looks like this:

```bash
# Stop the registry
docker stop <registry-container>

# Replace configuration with backup
cp config.yml.backup /etc/docker/registry/config.yml

# If needed, restore data
tar -xzf registry-data-backup.tar.gz -C /

# Restart the registry
docker start <registry-container>

# Verify
curl -v http://localhost:5000/v2/
docker push localhost:5000/alpine:test
```

If the rollback fails, do not leave the registry in an unknown state. Go back to the last known good configuration and data, even if it means temporarily losing recent changes. Avoid making multiple changes at once; this reduces the chance of an unrecoverable state.

## Common Pitfalls and How to Avoid Them

Even experienced operators fall into these traps. Recognizing them early saves hours of debugging.

### 1. Editing Configuration Without a Backup

**Why it happens:** Under pressure, someone edits `config.yml` directly and restarts the registry, then discovers the new setting breaks everything.

**How to avoid:** Always copy the config to a backup file before editing. Better, keep the config in version control and only deploy tested changes from the repository.

**How to recover:** Restore the backup immediately, restart, and verify.

### 2. Ignoring Disk Space Until It Fills Up

**Why it happens:** Registries grow silently, and a full disk causes push failures and sometimes data corruption.

**How to avoid:** Set up monitoring with alerts at 70% and 85% disk usage. Schedule regular garbage collection with the registry's built-in tool.

**How to recover:** Free space by deleting unused images or expanding the volume. Then run garbage collection to clean up orphaned blobs.

### 3. Not Testing Client Connectivity After Network Changes

**Why it happens:** A firewall or security group change breaks connectivity, but the operator only looks at server-side logs.

**How to avoid:** Always test from a client machine with `nc -zv registry-host 5000` and a real `docker pull` before closing the incident.

**How to recover:** Roll back the network change or open the necessary port and protocol.

### 4. Forgetting to Set Forwarded Headers Behind a Proxy

**Why it happens:** TLS termination at a proxy without setting `X-Forwarded-Proto` leads to incorrect URLs in the registry response, breaking pulls.

**How to avoid:** In proxy config, always set `X-Forwarded-Proto` and `X-Forwarded-For`. Test with curl and inspect response headers.

**How to recover:** Add the missing header, restart the proxy, and verify with a pull.

### 5. Running Garbage Collection Without a Dry Run

**Why it happens:** The operator runs `garbage-collect` in an attempt to free space, but it deletes blobs that are still referenced due to a misconfiguration, causing manifest failures.

**How to avoid:** Always run with `--dry-run` first and inspect the output. Perform a full backup before a real run.

**How to recover:** Restore from backup or re-push missing images.

## Operations Checklist

Use this checklist during routine operations and troubleshooting. For each major item, we name a single accountable owner to avoid ambiguity when something goes wrong. Review the checklist monthly, or immediately after any incident.

| Category | Item to Verify | Command / Action | Owner | Review Frequency |
|----------|----------------|------------------|-------|------------------|
| Pre-troubleshooting | Confirm registry version and config file location | `docker inspect <registry-container> --format '{{.Config.Image}}'` | Priya Shah, Platform Engineer | Monthly |
| Pre-troubleshooting | Check storage backend and disk space | `df -h /var/lib/registry` | Priya Shah | Weekly |
| Pre-troubleshooting | Verify network connectivity from client | `nc -zv registry-host 5000` | Arjun Patel, DevOps Engineer | Monthly |
| Pre-troubleshooting | Review recent changes in Git history | `git log --oneline -5 config.yml` | Priya Shah | Monthly |
| Diagnostics | Inspect registry logs for errors | `docker logs --tail 100 <registry-container>` | Arjun Patel | Weekly |
| Diagnostics | Test API endpoint | `curl -v http://localhost:5000/v2/` | Priya Shah | Weekly |
| Diagnostics | Push and pull a test image | `docker push localhost:5000/alpine:test` | Arjun Patel | Monthly |
| Diagnostics | Check storage directory permissions | `ls -l /var/lib/registry/docker/registry/v2/repositories` | Priya Shah | Monthly |
| Recovery | After fix, re-run health checks | `curl -v http://localhost:5000/v2/` | Priya Shah | After every change |
| Recovery | Confirm push/pull works | `docker push localhost:5000/alpine:test` | Arjun Patel | After every change |
| Recovery | Monitor logs for stability | `docker logs -f <registry-container>` | Arjun Patel | After every change |
| Preventive | Schedule regular backups | Cron job or external tool | Priya Shah | Monthly |
| Preventive | Monitor disk usage and set alerts | Prometheus + Alertmanager | Arjun Patel | Weekly |
| Preventive | Keep registry and clients updated | `docker pull registry:2.8.3` | Priya Shah | Quarterly |
| Preventive | Test changes in staging | Staging registry instance | Arjun Patel | Before production change |

## Conclusion

Docker Registry troubleshooting is a skill that combines methodical investigation with practical command-line fluency. By following the layered approach in this guide, from environment inventory to safe recovery, you can resolve most issues without escalating to panic. Start with a narrow, measurable problem: for example, "push fails with 500 error," then apply the diagnostic steps in order. Use the checklist to keep your registry healthy and to share knowledge across the team. Document your fixes in a runbook, and practice recovery in a lab so that when a real incident occurs, the response is almost automatic. Remember, the goal is not just to fix the registry, but to prevent the same failure from happening twice.