## Intro

Docker Registry configuration mistakes can turn a routine deployment into a long debugging session. Operators often discover a misconfiguration only after a push fails, a pull times out, or a restart wipes out image data. The goal of this article is to move you from observed problem to verified result using concrete commands, expected output, failure signals, and recovery steps.

This guide is for developers, DevOps consultants, and technical startup teams who run or plan to run a self-hosted Docker Registry. It connects configuration mistakes, validation, rollback, and troubleshooting to the exact commands that expose them. The emphasis is on operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery paths.

A Docker Registry stores and distributes container images. Unlike Docker Hub, a self-hosted registry gives you control over storage, access, and network placement, but that control comes with responsibility. Misconfigured storage backends, authentication, TLS, or garbage collection can lead to data loss or downtime. This article covers those risks and gives you a workflow to detect and avoid them.

## Version and Environment Inventory

Before touching a registry configuration file, know exactly what you are running. Version differences in Docker Registry (the `registry:2` image) or in Docker Engine can change how configuration options are interpreted.

Start with read-only observations:

```bash
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
```

Expected output shows the container name, uptime, and mapped ports. For example:

```
NAMES       STATUS          PORTS
registry    Up 3 hours      0.0.0.0:5000->5000/tcp
```

If the registry container is not listed, it is not running. Next, check the logs:

```bash
docker logs registry --tail 100
```

Look for startup errors such as `level=error msg="configuration error: ..."` or `level=fatal msg="invalid configuration"`. These lines often name the exact file and line number.

To inspect mounts, networks, environment variables, or health status:

```bash
docker inspect registry
```

For Compose projects, use equivalent commands:

```bash
docker compose ps
docker compose logs -f registry
docker compose exec registry sh
```

The last command gives you a shell inside the container without rebuilding the image.

### Where Data Lives: Volumes vs Bind Mounts

A common mistake is assuming the registry data survives a container recreation when it actually lives in the container filesystem. Check the storage configuration in the registry's `config.yml` (usually mounted at `/etc/docker/registry/config.yml`).

Example of a correct storage section using a named volume:

```yaml
storage:
  filesystem:
    rootdirectory: /var/lib/registry
```

And in Compose:

```yaml
services:
  registry:
    image: registry:2
    volumes:
      - registry_data:/var/lib/registry
volumes:
  registry_data:
```

Here `registry_data` is a Docker named volume managed by Docker. It persists across container rebuilds and is the recommended setup for production.

A bind mount example:

```yaml
volumes:
  - ./data:/var/lib/registry
```

This maps a host directory to the container path. It is easy to inspect but can cause permission problems if the container runs as a non-root user and the host directory is not writable. It also breaks portability if the path does not exist on another machine.

### Restart Test for Data Persistence

To verify your storage setup, perform a destructive restart test in a staging environment:

```bash
# Stop and remove the container
docker stop registry && docker rm registry

# Recreate it with the same volume
# (assuming you used a Compose file or the original docker run command)
docker run -d --name registry -p 5000:5000 -v registry_data:/var/lib/registry registry:2

# Push a test image, then check if it is still there after another restart
docker pull alpine:latest
docker tag alpine:latest localhost:5000/test-alpine:1.0
docker push localhost:5000/test-alpine:1.0
# Restart container
docker restart registry
# List tags in the registry using the API
curl -s http://localhost:5000/v2/test-alpine/tags/list
```

Expected output:

```json
{"name":"test-alpine","tags":["1.0"]}
```

If the tag list is empty or the curl command fails, the data was not persisted correctly.

## Safe Configuration Path

Changing the registry configuration requires a structured approach to avoid taking down image pulls or pushes for your team.

### Read the Current Configuration

First, extract the current configuration file from the running container:

```bash
docker cp registry:/etc/docker/registry/config.yml ./config.yml
```

Or if you use a bind mount, edit the file directly. Always make a backup:

```bash
cp config.yml config.yml.bak.$(date +%Y%m%d)
```

### Understand the Configuration File

The Docker Registry uses a YAML file with several top-level sections. Common ones include:

- `version`: usually `0.1`
- `log`: logging configuration
- `storage`: storage backend (filesystem, S3, Azure, etc.)
- `auth`: authentication method (e.g., htpasswd, token)
- `http`: TLS settings, address, and secret
- `health`: health check endpoints

A minimal production-ready configuration with TLS and authentication:

```yaml
version: 0.1
log:
  level: info
  fields:
    service: registry
storage:
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :443
  tls:
    certificate: /certs/domain.crt
    key: /certs/domain.key
auth:
  htpasswd:
    realm: basic-realm
    path: /auth/htpasswd
```

This assumes you have mounted certificates and htpasswd files. The registry fails to start if these paths are wrong.

### Make One Scoped Change at a Time

Suppose you need to increase the log level to `debug` for troubleshooting. Change only the `log.level` field:

Before:
```yaml
log:
  level: info
```

After:
```yaml
log:
  level: debug
```

Then restart the registry and watch the logs. If the change causes unexpected errors, revert immediately.

### Validate the Configuration Before Applying

The registry image does not include a `validate` command, but you can check the syntax using a YAML parser in a temporary container:

```bash
docker run --rm -v $(pwd)/config.yml:/config.yml -it yamllint yamllint /config.yml
```

Or use Python if available:

```bash
docker run --rm -v $(pwd)/config.yml:/config.yml python:3-alpine sh -c "python -c 'import yaml; yaml.safe_load(open(\"/config.yml\"))'"
```

If the command exits with an error, fix the YAML syntax before restarting.

### Use Environment Variables for Secrets

Never hardcode secrets in the configuration file. The registry supports environment variable substitution in the config. For example, use the `REGISTRY_STORAGE_S3_ACCESSKEY` environment variable for S3 storage.

For htpasswd authentication, do not put the password in YAML. Mount the htpasswd file and set permissions:

```bash
docker run -d --name registry \
  -p 443:443 \
  -v /path/to/certs:/certs \
  -v /path/to/auth:/auth \
  -e REGISTRY_AUTH=htpasswd \
  -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  registry:2
```

Use a secrets manager or Docker secrets if available.

## Verification and Diagnostics

After applying a configuration change, verify the registry behaves as expected. This section covers key checks.

### Health Endpoint

Registry v2 provides a health check endpoint. By default, it is at `/debug/health` but you can configure a custom path. With the default configuration, run:

```bash
curl -v http://localhost:5000/debug/health
```

Expected output:

```json
{}
```

And HTTP status 200. If the service is unhealthy, you may get a non-200 status or a connection error.

### API Version Check

Confirm the registry API version:

```bash
curl -s http://localhost:5000/v2/
```

Expected output includes `{}` with a 200 status. If authentication is enabled, you get a 401 with a `WWW-Authenticate` header.

### Test Push and Pull

Create a small test image and push it:

```bash
docker pull busybox:latest
docker tag busybox:latest localhost:5000/busybox:test
docker push localhost:5000/busybox:test
```

If the push succeeds, you see layer upload progress. Then pull the image on another machine or after deleting local copy:

```bash
docker rmi localhost:5000/busybox:test
docker pull localhost:5000/busybox:test
```

This validates that the registry is not only accepting pushes but also serving pulls.

### Check Logs for Errors

After any operation, check the logs:

```bash
docker logs registry --tail 50
```

Look for lines with `level=error` or `level=warning`. Common errors include:

- `level=error msg="response completed with error" err.code=unknown err.message="blob unknown to registry"`: This indicates a missing blob, often due to corrupted storage or manual deletion.
- `level=error msg="error serving error json: invalid character ..."`: Usually a problem with the registry's storage or configuration.

### Monitor Metrics

If you enabled Prometheus metrics (via `http.debug` or the `metrics` section), scrape `http://localhost:5000/metrics`. Check for increasing counters like `registry_http_requests_total` and error counters.

## Failure Modes and Recovery

Even with careful planning, failures happen. This section covers common failure modes and how to recover.

### Storage Backend Failure

If the filesystem rootdirectory becomes unwritable (disk full, permissions), the registry returns 500 errors on push. You can simulate this by making the directory read-only:

```bash
chmod 555 /var/lib/registry
```

Then try a push. The registry logs show `level=error msg="error writing blob"`. To recover:

```bash
chmod 755 /var/lib/registry
```

Check disk space with `df -h` and clean up if needed.

### Authentication Misconfiguration

If htpasswd file path is wrong, the registry starts but denies all authenticated requests or crashes. If the registry fails to start due to a missing htpasswd file, logs show:

```
level=fatal msg="open /auth/htpasswd: no such file or directory"
```

Fix by correcting the mount or creating the file:

```bash
docker run --entrypoint htpasswd httpd:2 -Bbn user password > htpasswd
```

Then mount it properly.

### TLS Certificate Problems

If the certificate is expired or does not match the hostname, clients get `x509: certificate signed by unknown authority` or `x509: certificate is valid for ... not ...`. The registry itself might start but clients fail to connect. To recover, replace the certificate files and restart the registry. Always set a reminder for certificate renewal.

### Garbage Collection Gone Wrong

Running garbage collection without stopping the registry can lead to corruption. The correct procedure is:

```bash
# Stop registry
docker stop registry
# Run garbage collection in a temporary container with the same storage volume
docker run --rm -v registry_data:/var/lib/registry registry:2 garbage-collect /etc/docker/registry/config.yml
# Restart registry
docker start registry
```

If the registry was running during GC, you may see blob missing errors later. In that case, you may need to restore from backup or re-push the affected images.

### Data Loss from Wrong Volume Mount

If the data directory was not mounted, after container recreation all images are gone. The recovery is to identify which images are missing and re-push them from CI or local caches. To avoid this, always run the restart test described earlier.

## Common Mistakes and How to Avoid Them

### 1. Running Without Persistent Storage

**Why it happens**: Developers often start a registry with a simple `docker run -d -p 5000:5000 registry:2` for testing and forget to add a volume.

**How to avoid**: Always define a volume or bind mount. Use Compose for reproducibility.

**Recovery**: If you have already lost data, re-push images from build pipelines.

### 2. Exposing the Registry Without TLS

**Why it happens**: For internal use, teams skip TLS to save effort. But Docker clients require TLS unless you configure insecure registries.

**How to avoid**: Use a trusted certificate even internally. If you must use HTTP, explicitly add the registry to `insecure-registries` in Docker daemon configuration.

**Recovery**: If clients can't connect, add the registry to insecure registries or set up TLS.

### 3. Hardcoding Secrets in Configuration Files

**Why it happens**: Quick setup scripts often include passwords in YAML.

**How to avoid**: Use environment variable substitution or Docker secrets. Keep configuration files in version control without secrets.

**Recovery**: Rotate the exposed secrets immediately.

### 4. Not Monitoring Disk Usage

**Why it happens**: Teams assume storage will be managed automatically.

**How to avoid**: Set up alerts on disk space and registry metrics. Periodically run garbage collection.

**Recovery**: Expand disk or clean up unused images.

### 5. Modifying Configuration Without Backup

**Why it happens**: Changes are made directly on the server without versioning.

**How to avoid**: Store configuration in version control. Always back up before manual edits.

**Recovery**: Restore from backup or version control.

### 6. Ignoring Registry Logs

**Why it happens**: Logs are considered noisy or inaccessible.

**How to avoid**: Centralize logs and set up alerts on error patterns.

**Recovery**: When issues arise, start by checking logs; they often contain the exact fix.

## Operations Checklist

Use this checklist before and after any registry configuration change. Each item includes a responsible role and a review frequency.

| Checklist Item | Owner | Frequency / Trigger |
|----------------|-------|---------------------|
| Verify current registry version and configuration backup | DevOps Engineer (e.g., Priya Shah) | Every change |
| Confirm storage volume mounted and writable | DevOps Engineer | Every change |
| Validate YAML syntax | DevOps Engineer | Every change |
| Restart registry and check health endpoint | DevOps Engineer | Every change |
| Push and pull a test image | QA Engineer (e.g., Marcus Lee) | Every change |
| Check logs for errors | DevOps Engineer | Every change |
| Monitor disk space and metrics | Sysadmin (e.g., Anna Kowalski) | Weekly |
| Review access and authentication | Security Officer (e.g., Tom Okafor) | Monthly |
| Test restore from backup | DevOps Engineer | Quarterly |

All decisions that affect production registry configuration should be approved by the DevOps lead, and the outcome should be reviewed at the monthly operations meeting.

## Conclusion

Docker Registry configuration mistakes are avoidable when you follow a disciplined approach: inventory your environment, make scoped changes with backups, verify with concrete commands, and plan for recovery. The registry is a critical piece of your container infrastructure; a misconfiguration can block all image distribution.

Start with the low-risk verification steps in this guide: run the restart test, check the health endpoint, and push/pull a test image. Then extend your checks to storage, TLS, and authentication.

A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Keep this checklist handy and revisit it whenever your team changes how images are built and deployed.