## Intro

A private Docker Registry is a critical piece of infrastructure for many development teams. It stores and serves container images, and when it fails, it can block CI/CD pipelines, prevent deployments, and frustrate developers. This guide focuses on common Docker Registry errors and how to fix them with practical examples. We cover configuration mistakes, storage issues, authentication problems, and operational pitfalls. Each section provides concrete commands, expected outputs, and recovery steps.

This article is written for operators, DevOps engineers, and developers who manage a self-hosted Docker Registry (the `registry:2` image from Docker, often called Distribution). It assumes you have basic Docker and Linux command-line skills and access to the registry host. We emphasize a safe approach: observe before changing, limit the blast radius, always use placeholders instead of real secrets in examples, and verify after every fix. We will not cover managed registries like Docker Hub, Amazon ECR, or Google Artifact Registry, except occasionally for comparison. The goal is to help you quickly diagnose and resolve issues with your own registry.

## Version and Environment Inventory

Before troubleshooting, you need a clear picture of your registry setup. Knowing the exact version, deployment method, and storage backend is essential because errors and fixes differ across versions.

Start by finding the registry container and its version. Run:

```bash
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}"
```

Look for a container using an image like `registry:2` or a custom image. To get the exact version, execute inside the container:

```bash
docker exec <registry-container> registry --version
```

Expected output example:
```
registry github.com/docker/distribution v2.8.3+unknown
```

If you are running the registry under Docker Compose or Kubernetes, find the manifest or deployment file. For Compose, `docker compose ps` and `docker compose config` show the current state and effective configuration. For Kubernetes, use `kubectl get pods -l app=registry -o yaml` and `kubectl describe pod <pod-name>`.

Also, note how the registry is configured. The main configuration file is typically `/etc/docker/registry/config.yml` inside the container, but it may be mounted from the host. Inspect the container mounts:

```bash
docker inspect <registry-container> --format '{{ json .Mounts }}' | jq
```

This shows volumes and bind mounts, which are crucial for storage troubleshooting. For example:
```json
[
  {
    "Type": "bind",
    "Source": "/opt/registry/data",
    "Destination": "/var/lib/registry",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  }
]
```

In this case, data is stored on the host at `/opt/registry/data`. If this path is not persistent or has incorrect permissions, the registry will fail to write images.

Always capture the current state before making changes. Save the output of `docker inspect`, the config file, and logs to a temporary directory. This gives you a rollback point and helps in post-mortem analysis. For example:

```bash
mkdir -p /tmp/registry-debug-$(date +%Y%m%d-%H%M%S)
docker inspect <registry-container> > /tmp/registry-debug-*/inspect.json
docker logs <registry-container> --tail 200 > /tmp/registry-debug-*/logs.txt
```

Use placeholders for secrets in any command you share or document. Do not paste real passwords or tokens.

## Safe Configuration Path

Many registry errors stem from misconfiguration. The registry uses a YAML file, commonly `config.yml`. Let's examine a minimal configuration and then discuss common mistakes.

A basic configuration that uses the local filesystem for storage and allows anonymous pulls might look like this:

```yaml
version: 0.1
log:
  level: info
  fields:
    service: registry
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: :5000
```

If you need authentication, you'd add an `auth` section, often using htpasswd:

```yaml
auth:
  htpasswd:
    realm: basic-realm
    path: /auth/htpasswd
```

But a common error is mismatched paths between the config file and the volume mounts. For instance, if you change `rootdirectory` to `/var/lib/registry` but the volume is mounted at `/var/lib/registry`, ensure the mount is correct. If you use a different path, like `/data`, but the mount is at `/var/lib/registry`, the registry writes to the container layer, and data disappears on container recreation.

To safely change configuration:
1. Copy the current config file from the container or host.
```bash
docker cp <registry-container>:/etc/docker/registry/config.yml /tmp/config.yml.backup
```
2. Make your changes in a test environment. If using Docker Compose, you can create a separate compose file for testing with a different port and storage.
3. Validate the YAML syntax. Use `docker run --rm -v /tmp/config.yml:/config.yml -it registry:2 registry serve /config.yml` but it will start serving; instead, you can use a YAML linter like `yamllint` on the host. Inside the container, you can use `python -c 'import yaml, sys; yaml.safe_load(sys.stdin)' < config.yml` if Python is available.
4. Apply the change by recreating the container with the new config (mount the file read-only).
5. Monitor logs: `docker logs -f <registry-container>`.

For configuration changes, always keep the blast radius small. Change one setting at a time, and document the expected outcome. If the registry fails to start, revert to the backup config and investigate.

A practical example: enabling the delete API to allow removing images. Many users hit "Method Not Allowed" errors when trying to delete an image because the delete feature is not enabled by default in the config. The error message from a client like `docker push` or `curl` might be:

```
HTTP 405 Method Not Allowed
```

The fix is to set `delete.enabled: true` under `storage` as shown above. After changing, restart the registry and test with a `curl -X DELETE` request to an image manifest. Note that enabling delete does not immediately free disk space; you need to run garbage collection (see Storage section).

## Verification and Diagnostics

After any change, you must verify that the registry behaves as expected. Start with health checks. The registry image includes a healthcheck if configured, but you can also use `curl` to the `/v2/` endpoint.

Run from the host or inside the container:

```bash
curl -v http://localhost:5000/v2/
```

Expected successful response:
```
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Docker-Distribution-Api-Version: registry/2.0

{}
```

If you get a 404 or connection refused, the registry is not listening properly or the port mapping is wrong. Check `docker port <registry-container>` and `docker logs` for errors like `listen tcp :5000: bind: address already in use`.

Another common diagnostic is checking image push and pull. Use a small test image, tag it to your local registry, and push:

```bash
docker pull alpine:latest
docker tag alpine:latest localhost:5000/alpine:test
docker push localhost:5000/alpine:test
```

Watch the output for errors. A successful push ends with a digest line. If authentication is enabled, you need to `docker login localhost:5000` first.

For deeper diagnostics, inspect the registry's API with `curl` to list repositories and tags:

```bash
curl -s http://localhost:5000/v2/_catalog
curl -s http://localhost:5000/v2/alpine/tags/list
```

Expected output:
```json
{"repositories":["alpine"]}
{"name":"alpine","tags":["test"]}
```

If you see errors, check the registry logs. Increase log level to `debug` in the config temporarily for more verbose output. Remember to revert to `info` after troubleshooting.

## Failure Modes and Recovery

Here are specific failure modes with their fixes.

### 1. `dial tcp: lookup registry on 127.0.0.53:53: no such host` when pulling from a remote registry

This error occurs when using a registry as a pull-through cache or when the registry tries to contact a remote registry but DNS resolution fails. Check the registry's network configuration, especially DNS settings. If running in Docker, ensure the container has proper DNS. You can specify DNS in the Docker run command with `--dns`. Alternatively, if using a proxy cache, ensure the `remoteurl` in the config is correct and accessible.

### 2. `manifest unknown` error on pull

When you push an image and then someone else tries to pull it, they might get `manifest unknown`. This often happens when the registry is configured with multiple storage backends (like S3 and filesystem) and the manifest is stored in one but not the other, or when the same repository name is used with different casing or tags. Verify consistency: check `docker tag` and ensure the repository name matches exactly. Also check if the registry is behind a load balancer that is routing to different instances without shared storage.

### 3. `blob upload unknown` during push

This error suggests that the registry cannot find the upload session, possibly because the session timed out or the registry restarted mid-upload. Increase the `http` timeouts in the config or ensure the client retries. A typical fix is to set larger values for `http` section:

```yaml
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
  timeouts:
    read: 900
    write: 900
    idle: 900
```
Restart and retry the push.

### 4. `denied: requested access to the resource is denied`

This is an authentication or authorization error. Check:
- Whether you are logged in (`docker login`).
- If the credentials are correct (htpasswd file).
- If the user has permission if using a more complex auth like token-based.

Use `curl -u username:password https://registry.example.com/v2/` to test. If the credentials are wrong, the server returns 401. Verify the htpasswd file is mounted correctly and the user exists: `docker exec <registry-container> htpasswd -B -C 10 /auth/htpasswd username` (note: `htpasswd` may not be in the image; you can use `registry:2` with `htpasswd` installed separately or use a tool on the host).

### 5. Out of disk space on the registry host

This is a silent failure: pushes may hang or fail with `io timeout`. Check disk usage: `df -h` on the host. If the registry's storage directory is on a full partition, you must free space. Often, the culprit is old unreferenced blobs. Run garbage collection (see Storage section). Also consider setting up monitoring and alerts for disk usage.

## Operations Checklist

Use this checklist to systematically troubleshoot and maintain your Docker Registry. It follows the observe-change-verify cycle.

| Step | Action | Command/Example | Expected Result | Owner | Frequency |
|------|--------|-----------------|-----------------|-------|-----------|
| 1 | Check registry container status | `docker ps -f name=registry` | Container running, healthy | DevOps Engineer | Daily (automated) |
| 2 | Check logs for errors | `docker logs registry --since 1h` | No critical errors | DevOps Engineer | Incident response |
| 3 | Verify storage mount | `docker inspect registry --format '{{ .Mounts }}'` | Correct bind mount or volume, RW | DevOps Engineer | Monthly |
| 4 | Test API health | `curl -f http://localhost:5000/v2/` | HTTP 200 | Monitoring system | Every 5 minutes |
| 5 | Push & pull test image | `docker push/pull` | Success | CI/CD pipeline | After every change |
| 6 | Check disk usage | `df -h /var/lib/registry` | Below 80% capacity | Infrastructure Admin | Weekly |
| 7 | Review authentication config | `cat /etc/docker/registry/config.yml` | No hardcoded secrets, correct paths | Security Officer | Quarterly |
| 8 | Run garbage collection (if needed) | `docker exec registry bin/registry garbage-collect /etc/docker/registry/config.yml` | Blobs deleted, space freed | DevOps Engineer | As needed |
| 9 | Backup registry data | `tar -czf registry-backup-$(date +%F).tar.gz /var/lib/registry` | Backup file created, tested | DevOps Engineer | Daily |
| 10 | Review security updates | `docker pull registry:2` | Latest stable image | DevOps Engineer | Monthly |

Each item has a single owner (not a team) to ensure accountability. Revisit this checklist quarterly or when the registry version changes.

## Storage and Data Integrity

Storage issues are among the most common and most dangerous. The registry stores images as blobs (layers) and manifests. If storage becomes corrupted or misconfigured, you may lose data.

**Filesystem vs. Object Storage:** The default is a local filesystem, but for production, object storage like S3, GCS, or Azure Blob is recommended for durability and scalability. Mixing storage types or changing storage backend without migrating data leads to `manifest unknown`.

**Permissions:** The registry process runs as user `root` in the container by default, but if you run it with a non-root user, ensure the storage directory is writable. A common error is `open /var/lib/registry/docker/registry/v2/repositories: permission denied` in the logs. Fix it by changing the ownership on the host directory: `sudo chown -R 1000:1000 /opt/registry/data` (assuming UID 1000 is the container user). Adjust the UID/GID as per your setup.

**Garbage Collection:** Over time, especially if you delete images or overwrite tags, unreferenced blobs accumulate. Run garbage collection periodically. The registry must be in read-only mode during collection to avoid corruption. Here is a safe procedure:

1. Stop the registry container or put it in read-only mode. You can do this by setting `storage.maintenance.readonly.enabled: true` in config and restarting, or simply stop the container.
2. Run garbage collection from the same image, mounting the same config and data:
```bash
docker run --rm -v /opt/registry/data:/var/lib/registry -v /opt/registry/config.yml:/etc/docker/registry/config.yml registry:2 bin/registry garbage-collect /etc/docker/registry/config.yml
```
Expected output lists the blobs marked for deletion.
3. Restart the registry normally.

**Backup:** Always back up the registry data before running garbage collection or making significant changes. The backup should include the storage directory and the config file. Test restores regularly.

## Security and Access Control

A misconfigured registry can expose private images or allow unauthorized pushes. Here are key mistakes and fixes:

- **Running without TLS:** In production, always use HTTPS. A registry without TLS requires Docker clients to be configured with `insecure-registries`, which is a security risk. Use a reverse proxy like Nginx or Caddy to terminate TLS and forward to the registry. A basic Nginx config:
```nginx
server {
    listen 443 ssl;
    server_name registry.example.com;

    ssl_certificate /etc/ssl/certs/registry.crt;
    ssl_certificate_key /etc/ssl/private/registry.key;

    location / {
        proxy_pass http://127.0.0.1:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}
```
- **Weak authentication:** If using htpasswd, generate bcrypt hashes. Avoid SHA or MD5. Use `htpasswd -B -C 12` for high cost. Never store plaintext passwords.
- **Exposing the registry to the internet without auth:** This allows anyone to push and pull. Always enable at least basic auth, and consider token-based auth for more granular control.
- **Using default secrets in config:** Avoid hardcoding tokens; use environment variables or secret mounts.

## Common Pitfalls and How to Avoid Them

Here are frequent mistakes that lead to registry failures:

1. **Not persisting storage** - Using the default container filesystem and then losing all images when the container is recreated. Always mount a volume for `/var/lib/registry` and ensure it is backed up.
2. **Misconfiguring the `rootdirectory` path** - Setting a path in config that does not match the mount point, causing data to be written to ephemeral storage.
3. **Ignoring log messages** - The registry logs are detailed and often tell you the exact error. Set log level to `info` or `debug` and check logs before guessing.
4. **Changing storage backend without migration** - Switching from filesystem to S3 and expecting old images to be available. You must migrate data or start fresh.
5. **Running garbage collection incorrectly** - Doing it while the registry is writable or using a different configuration can corrupt data. Always follow the safe procedure.
6. **Not monitoring disk and health** - A full disk can silently break pushes. Set up monitoring with Prometheus and alert on high usage.
7. **Overlooking security updates** - The registry image may have vulnerabilities. Regularly update to the latest patch version.

## Conclusion

Docker Registry errors can be frustrating, but with a systematic approach, they are manageable. Start by understanding your environment, make small, reversible changes, and verify after each step. Use the commands and examples in this guide to diagnose and fix common issues. Keep your configuration under version control, monitor the registry, and practice disaster recovery. By following these practices, you'll maintain a reliable and secure container image registry that supports your development workflow.

As a next step, pick one low-risk verification from this guide, such as testing the health endpoint or pushing a test image. Record the current state, run the check, and compare the result with the expected output. Then, review your storage and backup strategy to ensure your registry can survive failures.