## Intro

Docker logging drivers determine where container stdout and stderr streams are sent. When they fail, applications can appear healthy while logs disappear, disk usage grows without bound, or centralized logging pipelines silently lose events. This article provides a practical troubleshooting workflow for the most common Docker logging driver failures, with concrete commands, expected outputs, and recovery steps.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Each section targets a specific failure mode and includes a read-only diagnostic, the smallest justified change, and a verification command.

This guide assumes a Linux host with Docker Engine 20.10 or later and a basic understanding of container lifecycle. Commands are shown for both standalone Docker and Docker Compose projects. Where files or secrets appear, replace the placeholder values with your own environment data.

## Version and Environment Inventory

Before changing any logging configuration, capture the current state of the Docker daemon, the affected containers, and the host environment. The logging driver for a container is set at creation time, but the default driver for new containers is controlled by the daemon configuration. Start with read-only observations.

Run the following commands and record their output.

```bash
# Docker version and daemon configuration
docker version
sudo cat /etc/docker/daemon.json 2>/dev/null || echo "daemon.json not found"

# Effective default logging driver and options
docker info --format '{{.LoggingDriver}}'
docker info --format '{{json .Plugins.Log}}'

# Running containers with their configured logging driver
docker ps --format 'table {{.Names}}	{{.Status}}	{{.LogDriver}}'
```

Example output from a broken syslog setup:

```text
LoggingDriver: syslog
Containers:
NAMES      STATUS         LOGDRIVER
web        Up 2 hours     syslog
worker     Up 2 hours     syslog
db         Up 2 hours     json-file
```

If `daemon.json` exists, view it before making changes. A common misconfiguration is setting `log-driver` to a value that the daemon cannot load, which prevents containers from starting.

```bash
cat /etc/docker/daemon.json
```

Example bad configuration:

```json
{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://1.2.3.4:514"
  }
}
```

If the syslog server at `1.2.3.4` is unreachable, every new container will fail with an error like `failed to initialize logging driver: dial udp 1.2.3.4:514: connect: connection refused`. Existing containers are unaffected because their logging configuration was baked in at creation.

Check available logging plugins:

```bash
docker info | grep -A 5 "Logging Drivers"
```

Expected output includes `json-file`, `syslog`, `journald`, `gelf`, `fluentd`, `awslogs`, `splunk`, and others depending on your Docker version and installed plugins.

For data-driven logging setups, confirm where log files are written before making changes. The `json-file` driver writes to `/var/lib/docker/containers/<container-id>/<container-id>-json.log` by default. The `journald` driver sends logs to the systemd journal. The `syslog` driver sends to a remote or local syslog daemon. Use `docker inspect` to see the exact logging configuration for a container.

```bash
docker inspect --format '{{json .HostConfig.LogConfig}}' web
```

Example for a container with json-file logging and size rotation:

```json
{"Type":"json-file","Config":{"max-size":"10m","max-file":"3"}}
```

If the `Type` is `journald`, the container logs are not stored in Docker's directory, and `docker logs` may not work as expected.

A production-like local test should include a restart test: stop the container, recreate it, and confirm the application still emits logs to the expected destination. If logs disappear after a restart, the logging driver was likely not configured consistently across container recreations, or the daemon default was changed after the container was created.

## Safe Configuration Path

When a logging driver needs to be changed, follow a safe path: identify the component, verify prerequisites, make one scoped change, and verify the result. For container-level changes, you must recreate the container; for daemon-level changes, you must restart the Docker daemon (which restarts all running containers on most systems). Avoid changing both at once.

### Changing the logging driver for a single container

Suppose the `web` container currently uses `json-file`, but you need to switch to `syslog` for central collection. First, inspect existing configuration and note any custom options.

```bash
docker inspect --format '{{json .HostConfig.LogConfig}}' web
```

Then stop and remove the container, and recreate it with the new logging options. Use `docker run` with the `--log-driver` and `--log-opt` flags.

```bash
docker stop web
docker rm web

docker run -d --name web \
  --log-driver syslog \
  --log-opt syslog-address=tcp://logs.internal:514 \
  --log-opt syslog-facility=daemon \
  --log-opt tag="web/{{.Name}}/{{.ID}}" \
  myapp:latest
```

If you use Docker Compose, modify the service definition in `docker-compose.yml`:

```yaml
services:
  web:
    image: myapp:latest
    logging:
      driver: syslog
      options:
        syslog-address: "tcp://logs.internal:514"
        syslog-facility: "daemon"
        tag: "web/{{.Name}}/{{.ID}}"
```

Then recreate the service:

```bash
docker compose up -d web
```

Verify that logs are being sent to the syslog server. On the syslog server, check for incoming messages. If using a local syslog, inspect `/var/log/syslog` or `/var/log/messages` for entries tagged with `web`.

On the Docker host, you can also check that the container started without logging errors:

```bash
docker inspect --format '{{.State.Status}} {{.State.Error}}' web
```

Expected: `running` with no error. If the container is in `created` or `exited` state with an error mentioning the logging driver, the driver failed to initialize, often due to an unreachable syslog server or invalid options.

### Changing the default logging driver in daemon.json

Changing the daemon default affects only containers created after the change. Existing containers keep their original settings. To set the default logging driver to `json-file` with size-based rotation:

1. Edit `/etc/docker/daemon.json` (create if missing):

```json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
```

2. Restart the Docker daemon:

```bash
sudo systemctl restart docker
```

3. Verify the new default:

```bash
docker info --format '{{.LoggingDriver}}'
```

Expected: `json-file`. Then create a test container and inspect its logging config:

```bash
docker run -d --name logtest alpine sleep infinity
docker inspect --format '{{json .HostConfig.LogConfig}}' logtest
```

Expected output includes `"Type":"json-file"` with `"max-size":"10m"` and `"max-file":"3"`. If daemon restart fails, validate the JSON file with `python -m json.tool /etc/docker/daemon.json`. A common mistake is a trailing comma.

Never change `daemon.json` without a rollback plan. Before editing, copy the file:

```bash
sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.bak
```

If the daemon fails to start after the change, restore the backup and restart again. Keep the backup until you have verified that all new containers start and log correctly.

## Verification and Diagnostics

After a logging configuration change, verify that logs are flowing as expected. The first check is `docker logs` for containers using the `json-file` or `journald` drivers. For `syslog` or `fluentd`, `docker logs` may be empty because logs are sent to an external destination.

### Reading logs for json-file driver

For a container using `json-file`:

```bash
docker logs --tail 50 web
```

If this returns nothing but the application is generating output, check the log file on disk. First, get the container ID:

```bash
docker inspect --format '{{.Id}}' web
```

Then examine the log file (full ID is needed):

```bash
sudo ls -lh /var/lib/docker/containers/<full-id>/<full-id>-json.log
```

If the file size is 0 and the application has been running, the process may be writing to a file inside the container instead of stdout. Check the container's process behavior:

```bash
docker exec web sh -c 'ls -l /proc/1/fd/1 /proc/1/fd/2'
```

Expected output shows symlinks to `/dev/null` or a file inside the container, meaning stdout is redirected. Fix the application to write logs to stdout and stderr, not to files.

### Testing syslog delivery

For `syslog` driver, verify that logs reach the syslog server. On the Docker host, check for connection errors in the container's runtime log (not the container logs). The Docker daemon logs contain errors about logging driver failures.

```bash
sudo journalctl -u docker.service | grep -i syslog
```

Look for messages like:

```text
Failed to log message: write udp 192.168.1.10:514->logs.internal:514: connection refused
```

If you see connection refused, the syslog server is down or not listening. Confirm connectivity from the Docker host:

```bash
nc -vz logs.internal 514
```

If the port is closed, fix the syslog server or adjust the address. If the port is open but messages are not appearing, check the syslog server's configuration for facility and severity filters. The default facility is `daemon`, and the default severity is `info`. Some syslog servers drop messages with facility `daemon` if they only expect `local0`.

To change facility for a running container, you must recreate it. Use `--log-opt syslog-facility=local0` in `docker run` or the Compose equivalent.

### Analyzing journald logs

If using `journald`, `docker logs` may show logs, but the journal may also contain duplicate or missing entries. View logs from the journal for a specific container:

```bash
sudo journalctl CONTAINER_NAME=web
```

If the container name is not set as a journal field, use the container ID:

```bash
sudo journalctl CONTAINER_ID=$(docker inspect --format '{{.Id}}' web)
```

Check the logging driver options for `tag` and `labels`. The `tag` option adds a custom identifier to each log entry. Set it to include the container name or another unique field.

If journald logs are missing, check the journald rate limit. The default `RateLimitBurst` in `/etc/systemd/journald.conf` may drop logs when a container produces a high volume. Increase the limit and restart `systemd-journald`:

```config
RateLimitBurst=10000
RateLimitInterval=1s
```

Then restart both journald and the container.

## Failure Modes and Recovery

This section outlines common failure scenarios for Docker logging drivers, their symptoms, diagnosis, and recovery steps.

### Failure: Container fails to start with "failed to initialize logging driver"

**Symptom:** `docker run` or `docker compose up` returns an error like:

```text
docker: Error response from daemon: failed to initialize logging driver: dial tcp 10.0.0.5:514: connect: connection refused.
```

**Diagnosis:**

1. Check the configured logging driver and address:

```bash
docker inspect --format '{{json .HostConfig.LogConfig}}' <container-name-or-id>
```

If the container failed to create, inspect the command you used or the Compose file.

2. Verify connectivity to the logging endpoint:

For TCP syslog:

```bash
nc -vz 10.0.0.5 514
```

For UDP syslog (connectionless, so `nc -vz` may not work; use `nc -u`):

```bash
nc -u -w1 10.0.0.5 514 < /dev/null && echo "UDP sent"
```

Check that the syslog server is running and reachable.

**Recovery:**

- If the syslog server is temporarily down, you can start the container with a fallback logging driver that does not depend on the network, such as `json-file`.

```bash
docker run -d --name web --log-driver json-file myapp:latest
```

- To permanently change the driver, update the container configuration or Compose file. For daemon defaults, edit `daemon.json` and restart Docker.

- If you must use syslog but the server is down, start a local syslog receiver on the Docker host (e.g., `socat` or `rsyslog`) and point the container to `localhost:514`. When the remote server is back, switch the address.

### Failure: Docker logs shows no output but application is running

**Symptom:** `docker logs <container>` returns empty, yet the container is running and presumably producing output.

**Diagnosis:**

1. Check the container's logging driver:

```bash
docker inspect --format '{{.HostConfig.LogConfig.Type}}' <container>
```

If it is `syslog`, `fluentd`, `gelf`, or `awslogs`, `docker logs` will not display logs unless the driver supports the `--details` flag (it does not). Logs are sent to the external destination.

2. For `json-file`, verify that the application writes to stdout/stderr. Run a test command in a new container that definitely writes to stdout:

```bash
docker run --rm --log-driver json-file alpine echo "test log"
```

Then check `docker logs` for that container (it will be gone after `--rm`, so use `--name` and remove manually):

```bash
docker run -d --name logtest echo_image
docker logs logtest
```

If this works, the application is not writing to stdout. Capture the application's file descriptors as shown earlier.

**Recovery:**

- Modify the application to log to stdout/stderr. For many frameworks, set logging output to console. For Nginx, use `access_log /dev/stdout;` and `error_log /dev/stderr;`. For Python, use `logging.basicConfig(stream=sys.stdout)`.

- If you cannot change the application, use a sidecar container or a logging agent to tail files inside the container and forward them. For example, run a sidecar that mounts the container's log directory and reads files.

### Failure: Log files fill the disk

**Symptom:** The Docker host runs out of disk space, and `/var/lib/docker` is large.

**Diagnosis:**

1. Find the largest log files:

```bash
sudo find /var/lib/docker/containers -name "*-json.log" -exec du -h {} + | sort -rh | head -5
```

2. Check the logging options for the offending container:

```bash
docker inspect --format '{{json .HostConfig.LogConfig}}' <container>
```

If there is no `max-size` and `max-file`, the `json-file` driver grows without bound by default.

**Recovery:**

- For an existing container, you cannot change logging options without recreating. Recreate with size limits:

```bash
docker stop <container>
docker rm <container>
docker run -d --name <container> \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  <image>
```

- For all future containers, set the defaults in `daemon.json` as shown earlier.

- Immediately reclaim space by truncating large log files. This is safe for `json-file` logs, but it loses old entries:

```bash
sudo truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log
```

Do not delete the file; Docker may keep writing to the deleted inode.

### Failure: Syslog messages missing or not RFC3164 compliant

**Symptom:** Log messages appear in syslog but are malformed, missing fields, or not parsed by downstream systems.

**Diagnosis:**

- Check the syslog driver options:

```bash
docker inspect --format '{{json .HostConfig.LogConfig.Config}}' <container>
```

- Verify the syslog format. Docker sends RFC5424 format by default (`syslog-format=rfc5424`). Some legacy syslog servers expect RFC3164. Change the format:

```bash
--log-opt syslog-format=rfc3164
```

- Ensure the `tag` option is set to a unique identifier. The default tag is `{{.ID}}`, which is a long hex string. Use a more useful tag:

```bash
--log-opt tag="{{.Name}}/{{.ID}}"
```

**Recovery:** Recreate the container with corrected options. If the syslog server still misparses, enable debug logging on the syslog server to see raw messages. Adjust facility, severity, or format accordingly.

## Operations Checklist

Use this checklist before and after any logging configuration change.

| Step | Action | Command or File | Expected Result |
|------|--------|-----------------|-----------------|
| 1 | Record current Docker version and daemon logging defaults | `docker version`, `docker info --format '{{.LoggingDriver}}'` | Version 20.10+, driver name matches expectation |
| 2 | Inspect target container's current logging config | `docker inspect --format '{{json .HostConfig.LogConfig}}' <container>` | Shows current driver and options |
| 3 | Capture sample logs before change | `docker logs --tail 20 <container>` (if json-file) | Recent log lines visible |
| 4 | Backup daemon config if changing defaults | `sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.bak` | Backup file exists |
| 5 | Make one scoped change (container or daemon) | Edit Compose file or `daemon.json`; recreate container or restart daemon | No syntax errors |
| 6 | Verify new logging config | `docker inspect --format '{{json .HostConfig.LogConfig}}' <container>` | Driver and options match intended change |
| 7 | Generate test log entry | `docker exec <container> logger "test"` or equivalent | Entry appears at destination |
| 8 | Check for errors in daemon logs | `sudo journalctl -u docker.service --since "5 minutes ago" | grep -i "log"` | No errors related to logging driver |
| 9 | Confirm no unexpected restarts or state changes | `docker ps -a` | Containers in expected state |
| 10 | Document rollback plan and trigger | Write procedure in runbook | If new config fails, revert to backup and recreate containers |

For each row, replace the placeholder `<container>` with the actual container name. If a step fails, stop and investigate before proceeding. Do not make multiple changes at once; if the system breaks, you will not know which change caused it.

Example rollback for a daemon.json change that breaks container startup:

```bash
sudo systemctl stop docker
sudo cp /etc/docker/daemon.json.bak /etc/docker/daemon.json
sudo systemctl start docker
docker ps
```

## Conclusion

Docker logging driver issues are often the result of misconfiguration, network problems, or missing rotation limits. By following the structured workflow in this article—observe, make a scoped change, verify, and rollback if needed—you can resolve most failures without affecting production stability.

Start with the Version and Environment Inventory to capture the current state. Use the Safe Configuration Path to change drivers with minimal blast radius. Verify log delivery with concrete commands. When things go wrong, refer to the Failure Modes and Recovery section for step-by-step recovery.

Finally, adopt the Operations Checklist as a standard runbook for any logging changes. Document your environment-specific values, keep backups, and test changes in a staging environment first. With these practices, Docker logging drivers become a reliable part of your observability stack rather than a source of operational surprises.