Docker Logging Drivers Troubleshooting with Practical Examples

Intro

Docker logging drivers determine where container stdout and stderr streams are sent. When they fail, applications can appear healthy while logs disappear, disk usage grows without bound, or centralized logging pipelines silently lose events. This article provides a practical troubleshooting workflow for the most common Docker logging driver failures, with concrete commands, expected outputs, and recovery steps.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Each section targets a specific failure mode and includes a read-only diagnostic, the smallest justified change, and a verification command.

This guide assumes a Linux host with Docker Engine 20.10 or later and a basic understanding of container lifecycle. Commands are shown for both standalone Docker and Docker Compose projects. Where files or secrets appear, replace the placeholder values with your own environment data.

Version and Environment Inventory

Before changing any logging configuration, capture the current state of the Docker daemon, the affected containers, and the host environment. The logging driver for a container is set at creation time, but the default driver for new containers is controlled by the daemon configuration. Start with read-only observations.

Run the following commands and record their output.

# Docker version and daemon configuration
docker version
sudo cat /etc/docker/daemon.json 2>/dev/null || echo "daemon.json not found"

# Effective default logging driver and options
docker info --format '{{.LoggingDriver}}'
docker info --format '{{json .Plugins.Log}}'

# Running containers with their configured logging driver
docker ps --format 'table {{.Names}}	{{.Status}}	{{.LogDriver}}'

Example output from a broken syslog setup:

LoggingDriver: syslog
Containers:
NAMES      STATUS         LOGDRIVER
web        Up 2 hours     syslog
worker     Up 2 hours     syslog
db         Up 2 hours     json-file

If daemon.json exists, view it before making changes. A common misconfiguration is setting log-driver to a value that the daemon cannot load, which prevents containers from starting.

cat /etc/docker/daemon.json

Example bad configuration:

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://1.2.3.4:514"
  }
}

If the syslog server at 1.2.3.4 is unreachable, every new container will fail with an error like failed to initialize logging driver: dial udp 1.2.3.4:514: connect: connection refused. Existing containers are unaffected because their logging configuration was baked in at creation.

Check available logging plugins:

docker info | grep -A 5 "Logging Drivers"

Expected output includes json-file, syslog, journald, gelf, fluentd, awslogs, splunk, and others depending on your Docker version and installed plugins.

For data-driven logging setups, confirm where log files are written before making changes. The json-file driver writes to /var/lib/docker/containers/<container-id>/<container-id>-json.log by default. The journald driver sends logs to the systemd journal. The syslog driver sends to a remote or local syslog daemon. Use docker inspect to see the exact logging configuration for a container.

docker inspect --format '{{json .HostConfig.LogConfig}}' web

Example for a container with json-file logging and size rotation:

{"Type":"json-file","Config":{"max-size":"10m","max-file":"3"}}

If the Type is journald, the container logs are not stored in Docker's directory, and docker logs may not work as expected.

A production-like local test should include a restart test: stop the container, recreate it, and confirm the application still emits logs to the expected destination. If logs disappear after a restart, the logging driver was likely not configured consistently across container recreations, or the daemon default was changed after the container was created.

Safe Configuration Path

When a logging driver needs to be changed, follow a safe path: identify the component, verify prerequisites, make one scoped change, and verify the result. For container-level changes, you must recreate the container; for daemon-level changes, you must restart the Docker daemon (which restarts all running containers on most systems). Avoid changing both at once.

Changing the logging driver for a single container

Suppose the web container currently uses json-file, but you need to switch to syslog for central collection. First, inspect existing configuration and note any custom options.

docker inspect --format '{{json .HostConfig.LogConfig}}' web

Then stop and remove the container, and recreate it with the new logging options. Use docker run with the --log-driver and --log-opt flags.

docker stop web
docker rm web

docker run -d --name web \
  --log-driver syslog \
  --log-opt syslog-address=tcp://logs.internal:514 \
  --log-opt syslog-facility=daemon \
  --log-opt tag="web/{{.Name}}/{{.ID}}" \
  myapp:latest

If you use Docker Compose, modify the service definition in docker-compose.yml:

services:
  web:
    image: myapp:latest
    logging:
      driver: syslog
      options:
        syslog-address: "tcp://logs.internal:514"
        syslog-facility: "daemon"
        tag: "web/{{.Name}}/{{.ID}}"

Then recreate the service:

docker compose up -d web

Verify that logs are being sent to the syslog server. On the syslog server, check for incoming messages. If using a local syslog, inspect /var/log/syslog or /var/log/messages for entries tagged with web.

On the Docker host, you can also check that the container started without logging errors:

docker inspect --format '{{.State.Status}} {{.State.Error}}' web

Expected: running with no error. If the container is in created or exited state with an error mentioning the logging driver, the driver failed to initialize, often due to an unreachable syslog server or invalid options.

Changing the default logging driver in daemon.json

Changing the daemon default affects only containers created after the change. Existing containers keep their original settings. To set the default logging driver to json-file with size-based rotation:

1. Edit /etc/docker/daemon.json (create if missing):

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

1. Restart the Docker daemon:

sudo systemctl restart docker

1. Verify the new default:

docker info --format '{{.LoggingDriver}}'

Expected: json-file. Then create a test container and inspect its logging config:

docker run -d --name logtest alpine sleep infinity
docker inspect --format '{{json .HostConfig.LogConfig}}' logtest

Expected output includes "Type":"json-file" with "max-size":"10m" and "max-file":"3". If daemon restart fails, validate the JSON file with python -m json.tool /etc/docker/daemon.json. A common mistake is a trailing comma.

Never change daemon.json without a rollback plan. Before editing, copy the file:

sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.bak

If the daemon fails to start after the change, restore the backup and restart again. Keep the backup until you have verified that all new containers start and log correctly.

Verification and Diagnostics

After a logging configuration change, verify that logs are flowing as expected. The first check is docker logs for containers using the json-file or journald drivers. For syslog or fluentd, docker logs may be empty because logs are sent to an external destination.

Reading logs for json-file driver

For a container using json-file:

docker logs --tail 50 web

If this returns nothing but the application is generating output, check the log file on disk. First, get the container ID:

docker inspect --format '{{.Id}}' web

Then examine the log file (full ID is needed):

sudo ls -lh /var/lib/docker/containers/<full-id>/<full-id>-json.log

If the file size is 0 and the application has been running, the process may be writing to a file inside the container instead of stdout. Check the container's process behavior:

docker exec web sh -c 'ls -l /proc/1/fd/1 /proc/1/fd/2'

Expected output shows symlinks to /dev/null or a file inside the container, meaning stdout is redirected. Fix the application to write logs to stdout and stderr, not to files.

Testing syslog delivery

For syslog driver, verify that logs reach the syslog server. On the Docker host, check for connection errors in the container's runtime log (not the container logs). The Docker daemon logs contain errors about logging driver failures.

sudo journalctl -u docker.service | grep -i syslog

Look for messages like:

Failed to log message: write udp 192.168.1.10:514->logs.internal:514: connection refused

If you see connection refused, the syslog server is down or not listening. Confirm connectivity from the Docker host:

nc -vz logs.internal 514

If the port is closed, fix the syslog server or adjust the address. If the port is open but messages are not appearing, check the syslog server's configuration for facility and severity filters. The default facility is daemon, and the default severity is info. Some syslog servers drop messages with facility daemon if they only expect local0.

To change facility for a running container, you must recreate it. Use --log-opt syslog-facility=local0 in docker run or the Compose equivalent.

Analyzing journald logs

If using journald, docker logs may show logs, but the journal may also contain duplicate or missing entries. View logs from the journal for a specific container:

sudo journalctl CONTAINER_NAME=web

If the container name is not set as a journal field, use the container ID:

sudo journalctl CONTAINER_ID=$(docker inspect --format '{{.Id}}' web)

Check the logging driver options for tag and labels. The tag option adds a custom identifier to each log entry. Set it to include the container name or another unique field.

If journald logs are missing, check the journald rate limit. The default RateLimitBurst in /etc/systemd/journald.conf may drop logs when a container produces a high volume. Increase the limit and restart systemd-journald:

RateLimitBurst=10000
RateLimitInterval=1s

Then restart both journald and the container.

Failure Modes and Recovery

This section outlines common failure scenarios for Docker logging drivers, their symptoms, diagnosis, and recovery steps.

Failure: Container fails to start with "failed to initialize logging driver"

Symptom: docker run or docker compose up returns an error like:

docker: Error response from daemon: failed to initialize logging driver: dial tcp 10.0.0.5:514: connect: connection refused.

Diagnosis:

1. Check the configured logging driver and address:

docker inspect --format '{{json .HostConfig.LogConfig}}' <container-name-or-id>

If the container failed to create, inspect the command you used or the Compose file.

1. Verify connectivity to the logging endpoint:

For TCP syslog:

nc -vz 10.0.0.5 514

For UDP syslog (connectionless, so nc -vz may not work; use nc -u):

nc -u -w1 10.0.0.5 514 < /dev/null && echo "UDP sent"

Check that the syslog server is running and reachable.

Recovery:

• If the syslog server is temporarily down, you can start the container with a fallback logging driver that does not depend on the network, such as json-file.

docker run -d --name web --log-driver json-file myapp:latest

• To permanently change the driver, update the container configuration or Compose file. For daemon defaults, edit daemon.json and restart Docker.

• If you must use syslog but the server is down, start a local syslog receiver on the Docker host (e.g., socat or rsyslog) and point the container to localhost:514. When the remote server is back, switch the address.

Failure: Docker logs shows no output but application is running

Symptom: docker logs <container> returns empty, yet the container is running and presumably producing output.

Diagnosis:

1. Check the container's logging driver:

docker inspect --format '{{.HostConfig.LogConfig.Type}}' <container>

If it is syslog, fluentd, gelf, or awslogs, docker logs will not display logs unless the driver supports the --details flag (it does not). Logs are sent to the external destination.

1. For json-file, verify that the application writes to stdout/stderr. Run a test command in a new container that definitely writes to stdout:

docker run --rm --log-driver json-file alpine echo "test log"

Then check docker logs for that container (it will be gone after --rm, so use --name and remove manually):

docker run -d --name logtest echo_image
docker logs logtest

If this works, the application is not writing to stdout. Capture the application's file descriptors as shown earlier.

Recovery:

• Modify the application to log to stdout/stderr. For many frameworks, set logging output to console. For Nginx, use access_log /dev/stdout; and error_log /dev/stderr;. For Python, use logging.basicConfig(stream=sys.stdout).

• If you cannot change the application, use a sidecar container or a logging agent to tail files inside the container and forward them. For example, run a sidecar that mounts the container's log directory and reads files.

Failure: Log files fill the disk

Symptom: The Docker host runs out of disk space, and /var/lib/docker is large.

Diagnosis:

1. Find the largest log files:

sudo find /var/lib/docker/containers -name "*-json.log" -exec du -h {} + | sort -rh | head -5

1. Check the logging options for the offending container:

docker inspect --format '{{json .HostConfig.LogConfig}}' <container>

If there is no max-size and max-file, the json-file driver grows without bound by default.

Recovery:

• For an existing container, you cannot change logging options without recreating. Recreate with size limits:

docker stop <container>
docker rm <container>
docker run -d --name <container> \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  <image>

• For all future containers, set the defaults in daemon.json as shown earlier.

• Immediately reclaim space by truncating large log files. This is safe for json-file logs, but it loses old entries:

sudo truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log

Do not delete the file; Docker may keep writing to the deleted inode.

Failure: Syslog messages missing or not RFC3164 compliant

Symptom: Log messages appear in syslog but are malformed, missing fields, or not parsed by downstream systems.

Diagnosis:

• Check the syslog driver options:

docker inspect --format '{{json .HostConfig.LogConfig.Config}}' <container>

• Verify the syslog format. Docker sends RFC5424 format by default (syslog-format=rfc5424). Some legacy syslog servers expect RFC3164. Change the format:

--log-opt syslog-format=rfc3164

• Ensure the tag option is set to a unique identifier. The default tag is {{.ID}}, which is a long hex string. Use a more useful tag:

--log-opt tag="{{.Name}}/{{.ID}}"

Recovery: Recreate the container with corrected options. If the syslog server still misparses, enable debug logging on the syslog server to see raw messages. Adjust facility, severity, or format accordingly.

Operations Checklist

Use this checklist before and after any logging configuration change.

For each row, replace the placeholder <container> with the actual container name. If a step fails, stop and investigate before proceeding. Do not make multiple changes at once; if the system breaks, you will not know which change caused it.

Example rollback for a daemon.json change that breaks container startup:

sudo systemctl stop docker
sudo cp /etc/docker/daemon.json.bak /etc/docker/daemon.json
sudo systemctl start docker
docker ps

Conclusion

Docker logging driver issues are often the result of misconfiguration, network problems, or missing rotation limits. By following the structured workflow in this article—observe, make a scoped change, verify, and rollback if needed—you can resolve most failures without affecting production stability.

Start with the Version and Environment Inventory to capture the current state. Use the Safe Configuration Path to change drivers with minimal blast radius. Verify log delivery with concrete commands. When things go wrong, refer to the Failure Modes and Recovery section for step-by-step recovery.

Finally, adopt the Operations Checklist as a standard runbook for any logging changes. Document your environment-specific values, keep backups, and test changes in a staging environment first. With these practices, Docker logging drivers become a reliable part of your observability stack rather than a source of operational surprises.