Docker Logging Drivers Local Lab: A Practical Setup Guide with Examples

Intro

Docker logging drivers control how container logs are collected and stored. The default json-file driver writes logs locally, but other drivers can send logs to a remote system, add metadata, or reduce disk usage. Setting up a local lab lets you test driver behavior without touching production environments.

This guide walks through a complete Docker logging driver lab on a single Linux host. You will inspect the current logging setup, configure and test several drivers, learn how to rotate and read logs, and practice recovery when things go wrong. Every step includes commands you can run and the expected output, so you can verify each result before moving on.

The examples assume Docker Engine 24.0 or later, with the Docker CLI and Docker Compose v2 installed. Without systemd journal access or rsyslog, some drivers may not work as shown. We will cover the prerequisites for each driver.

Version and Environment Inventory

Before changing any configuration, record your Docker version, storage driver, and current logging setup. This gives you a rollback point and helps you spot incompatible options.

Run these commands and save the output:

docker version --format '{{.Server.Version}}'
docker info --format 'Storage Driver: {{.Driver}}'
docker info --format 'Logging Driver: {{.LoggingDriver}}'

Expected output on a fresh Docker installation may look like this:

24.0.7
Storage Driver: overlay2
Logging Driver: json-file

If your output differs, note it. For example, some cloud images set journald or syslog as the default driver. The lab assumes you can override the driver per container at runtime, which all modern Docker versions support.

Also check that you have permission to run Docker without sudo. If not, add your user to the docker group and log out and back in. This is not strictly required, but it avoids permission errors when reading log files.

Safe Configuration Path

We will configure each logging driver with a minimal set of options and then run a container that writes predictable log lines. This keeps the changes small and easy to undo.

Default json-file driver and log rotation

The json-file driver stores logs as JSON objects under /var/lib/docker/containers/<container-id>/<container-id>-json.log. Without rotation, this file can grow until the disk fills. Production best practice is to set max-size and max-file.

Start a container with explicit rotation settings:

docker run -d --name log-json \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  alpine sh -c 'while true; do echo "$(date) hello from json-file"; sleep 1; done'

This writes one line per second. To create enough data for rotation, you can temporarily lower max-size to 1k in a second container:

docker run -d --name log-rotate-test \
  --log-driver json-file \
  --log-opt max-size=1k \
  --log-opt max-file=3 \
  alpine sh -c 'while true; do echo "$(date) rotate me"; done'

After a minute or so, list the log files:

sudo ls -lh /var/lib/docker/containers/$(docker inspect -f '{{.Id}}' log-rotate-test)/

You should see multiple files ending in -json.log, -json.log.1, -json.log.2, and possibly a compressed .gz file, depending on your Docker version. This confirms rotation works.

Stop and remove the test containers before continuing:

docker rm -f log-json log-rotate-test

Syslog driver

The syslog driver sends logs to a syslog daemon. On most Linux hosts, rsyslog or syslog-ng is already running. If not, install rsyslog:

sudo apt update && sudo apt install -y rsyslog   # Debian/Ubuntu
sudo systemctl start rsyslog

Run a container using the syslog driver:

docker run -d --name log-syslog \
  --log-driver syslog \
  --log-opt syslog-address=udp://127.0.0.1:514 \
  --log-opt tag="docker/{{.Name}}" \
  alpine sh -c 'echo "syslog test from container" && sleep 60'

Check the host syslog:

sudo tail -n 5 /var/log/syslog | grep 'syslog test'

Expected output includes a line with the tag docker/log-syslog and the message. If you see nothing, confirm rsyslog is listening on UDP 514:

sudo ss -ulnp | grep 514

The output should show a process like rsyslogd listening on 0.0.0.0:514. If not, edit /etc/rsyslog.conf and uncomment the lines for UDP syslog reception, then restart rsyslog.

Stop the container after verifying:

docker rm -f log-syslog

Journald driver

The journald driver writes logs to the systemd journal. This requires the Docker daemon to run on a host with systemd. Check if journald is available:

systemctl is-active systemd-journald

Expected output: active. If it says inactive or failed, you need to start it with sudo systemctl start systemd-journald.

Run a container with the journald driver:

docker run -d --name log-journald \
  --log-driver journald \
  --log-opt tag="docker-journald-test" \
  alpine sh -c 'echo "journald test" && sleep 60'

Read the log from the journal:

journalctl CONTAINER_NAME=log-journald --no-pager

Expected output includes the journald test line along with systemd metadata. If you see a permission error, either run journalctl with sudo or ensure your user is in the systemd-journal group.

Fluentd driver

The fluentd driver sends logs to a Fluentd instance. This is useful when you already run Fluentd for log aggregation. For a quick lab, start a Fluentd container that listens on TCP port 24224 and writes forwarded logs to stdout.

Create a minimal Fluentd config file fluent.conf:

<source>
  @type forward
  port 24224
</source>
<match **>
  @type stdout
</match>

Run Fluentd:

docker run -d --name fluentd-lab \
  -p 24224:24224 \
  -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf \
  fluent/fluentd:v1.16-1

Wait a few seconds, then run a container that uses the Fluentd driver:

docker run --rm --name log-fluentd \
  --log-driver fluentd \
  --log-opt fluentd-address=127.0.0.1:24224 \
  alpine echo "fluentd test from container"

Check Fluentd's own logs to see the forwarded entry:

docker logs fluentd-lab | tail -n 3

Expected output includes a line containing "fluentd test from container" in the Fluentd event format. If the connection fails, check that Fluentd is listening:

docker logs fluentd-lab | grep 'listening'

You should see a line like listening fluent socket on 0.0.0.0:24224.

Stop Fluentd when done:

docker rm -f fluentd-lab

Gelf driver (optional)

The gelf driver sends logs to a Graylog or other GELF-compatible endpoint. For a lab, you can use a simple UDP listener to test the format. Here is a quick Python UDP server that prints received datagrams:

docker run --rm -d --name gelf-listener \
  -p 12201:12201/udp \
  python:3.11-alpine python -c "
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('0.0.0.0', 12201))
while True:
    data, addr = s.recvfrom(65535)
    print(data.decode(), flush=True)
"

Now run a container that logs to GELF UDP:

docker run --rm --name log-gelf \
  --log-driver gelf \
  --log-opt gelf-address=udp://127.0.0.1:12201 \
  alpine echo "gelf test"

Check the listener output:

docker logs gelf-listener

You should see a GELF JSON payload containing the gelf test message. If not, verify the UDP port mapping with docker port gelf-listener.

Clean up:

docker rm -f gelf-listener

Verification and Diagnostics

After configuring a driver, you must confirm that logs are actually flowing and that the driver is not silently dropping messages.

Verify driver selection

Use docker inspect to see which logging driver a container uses and its options:

docker inspect -f '{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}' <container-name>

For example, for the log-json container you ran earlier, the output would be:

json-file map[max-file:3 max-size:10m]

Run this for each container after creation to catch typos in driver names or options.

Read logs with docker logs

docker logs works for most drivers, but it may behave differently. The json-file and journald drivers support docker logs natively. The syslog and fluentd drivers do not return logs via docker logs; you must query the external destination.

Test this behavior:

docker logs log-syslog

Expected: an error like the logging driver "syslog" does not support the "logs" subcommand or no output at all. This is a known limitation. Plan your diagnostics accordingly: for remote drivers, check the receiving endpoint instead of docker logs.

Inspect log files on disk

For the json-file driver, you can directly read the log file to see the JSON structure:

sudo cat /var/lib/docker/containers/$(docker inspect -f '{{.Id}}' <container>) /<container>-json.log | head -n 2

Each line is a JSON object with keys log, stream, and time. For example:

{"log":"hello from json-file\n","stream":"stdout","time":"2025-03-20T15:04:05Z"}

This is useful for scripting or when debugging log parsing tools.

Check driver metrics

Docker exposes logging driver errors in the daemon logs. To see if the daemon is having trouble with a driver, check the Docker daemon journal:

sudo journalctl -u docker.service --since "10 minutes ago" | grep -i 'log'

Look for messages like Failed to log message or driver-specific errors. For the syslog driver, a common error is connection refused if the syslog address is wrong.

Failure Modes and Recovery

Logging drivers can fail in ways that are not immediately obvious. Knowing the common failure modes helps you recover quickly and design robust logging setups.

Failure: json-file logs fill the disk

Symptom: Containers stop writing logs, or the Docker daemon reports no space left on device. The root cause is usually no log rotation.

Recovery:

1. Find the largest log files:

   sudo find /var/lib/docker/containers -name '*-json.log' -size +100M -ls

1. Truncate oversized logs immediately to free space:

   sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

1. Prevent recurrence by setting max-size and max-file on the container. You can also set global defaults in the daemon configuration file /etc/docker/daemon.json:

   {
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "10m",
       "max-file": "3"
     }
   }

Restart Docker after changing daemon.json:

   sudo systemctl restart docker

Note: This restart will stop all running containers. Do this during a maintenance window.

Failure: Syslog driver cannot connect

Symptom: Container starts, but no logs appear in /var/log/syslog. The Docker daemon log shows connection refused for the syslog address.

Recovery:

1. Verify the syslog daemon is running and listening on the expected socket:

   sudo ss -ulnp | grep 514

1. If nothing is listening, start rsyslog:

   sudo systemctl start rsyslog

1. If running a containerized syslog server, ensure its port is published correctly. For the earlier example, the syslog receiver used udp://127.0.0.1:514. If you run the receiver in a container, you need to publish UDP port 514 and use the host IP.

1. To avoid future connection failures, add a retry and a longer timeout to the driver options:

   --log-opt syslog-address=udp://127.0.0.1:514 \
   --log-opt syslog-retries=5 \
   --log-opt syslog-timeout=10s

Failure: Journald driver drops logs on restart

Symptom: Logs from a container are missing after the container restarts. The journald driver tags logs with the container ID, not the name. If the container is removed and recreated, the ID changes, and older logs are harder to find.

Recovery:

1. Use the container ID to query logs:

   journalctl CONTAINER_ID=<full-container-id>

1. Add a custom tag that includes the container name for easier filtering:

   --log-opt tag="myapp-{{.Name}}"

Then query with:

   journalctl SYSLOG_IDENTIFIER=myapp-<name>

1. In Compose files, you can set a stable container name with container_name to make the tag predictable.

Failure: Fluentd driver buffers fill up

Symptom: Container logs are delayed, or the Fluentd driver reports buffer overflow. The Fluentd driver buffers logs in memory when the Fluentd endpoint is slow or unreachable. If the buffer fills, the container may block or drop logs.

Recovery:

1. Check the Docker daemon log for Fluentd errors:

   sudo journalctl -u docker.service | grep fluentd

1. Increase the Fluentd buffer limit if you expect bursts:

   --log-opt fluentd-buffer-limit=64m

1. Ensure the Fluentd endpoint has enough capacity. Tune Fluentd's flush intervals and buffer sizes on the server side.

1. For production, use a dedicated log collector with proper queueing, such as Fluentd with a file buffer.

Failure: Gelf driver loses logs over UDP

Symptom: Some logs never appear at the GELF endpoint. UDP is connectionless and does not guarantee delivery. Under load or network issues, packets can be lost.

Recovery:

1. Use TCP instead of UDP for GELF if your endpoint supports it:

   --log-opt gelf-address=tcp://127.0.0.1:12201

1. Increase the gelf-compression-level to reduce bandwidth, but this does not prevent UDP loss. For reliable delivery, use a driver that supports TCP or a message queue.

1. Monitor the endpoint for missing sequence numbers or gaps in logs.

Common Pitfalls and How to Avoid Them

Several mistakes recur when working with Docker logging drivers in a lab or production. Being aware of them saves time and prevents data loss.

Ignoring driver support for docker logs

Many users assume docker logs works for all drivers. It does not. The json-file, journald, and local drivers support docker logs. The syslog, fluentd, gelf, and awslogs drivers do not. If you rely on docker logs for debugging, choose a compatible driver.

How to avoid: Before selecting a driver, check the Docker documentation for its docker logs support. Test with a small container early.

Not setting log rotation for json-file

The default json-file driver has no size limit. Over months, logs can consume all disk space, causing Docker and other services to fail.

How to avoid: Always set max-size and max-file on every container using json-file. Better yet, set global defaults in the daemon configuration.

Using bind mounts for log files without proper permissions

When mounting a directory from the host to store log files, the container user may not have write permission, causing the application to fail to start.

How to avoid: Check the container's user and the mount permissions. Use a named volume instead:

docker volume create app-logs
docker run -d --name app \
  -v app-logs:/var/log/app \
  --log-driver json-file \
  myapp

Named volumes are managed by Docker and handle permissions more gracefully.

Mixing logging driver options with daemon-level settings

If you set a global logging driver in daemon.json but also pass --log-driver on the container, the container option overrides the global. This can lead to inconsistent logging.

How to avoid: Document the intended driver per service. Use explicit container-level options for special cases only. Verify with docker inspect after startup.

Forgetting to publish UDP ports for remote logging receivers

When running a syslog or GELF receiver in a container, you must publish its UDP port. Omitting -p 514:514/udp results in silent failure; the sending container cannot reach the receiver on localhost.

How to avoid: Always check port mappings with docker port <receiver-container>. Test connectivity with nc -u <host> <port> before relying on the setup.

Not cleaning up lab containers and files

After testing, orphan containers and leftover config files can interfere with later experiments or consume resources.

How to avoid: Keep a cleanup script. For example:

docker rm -f log-json log-syslog log-journald log-fluentd log-gelf
rm -f fluent.conf

Regularly run docker system prune in a lab environment, but be careful in shared or production hosts.

Operations Checklist

Use this checklist whenever you deploy or modify a Docker logging driver in a lab or production-like environment.

Each item has a single owner who is accountable for execution and follow-up. The revisit frequency ensures the checklist remains relevant as infrastructure evolves.

Configuration example using Docker Compose

A complete Compose file for a web service with json-file logging and rotation:

version: '3.8'
services:
  web:
    image: nginx:1.25
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    deploy:
      resources:
        limits:
          memory: 128M

For a service that sends logs to a Fluentd sidecar, you can define:

services:
  app:
    image: myapp:latest
    logging:
      driver: fluentd
      options:
        fluentd-address: "fluentd:24224"
        tag: "app.{{.Name}}"
  fluentd:
    image: fluent/fluentd:v1.16-1
    ports:
      - "24224:24224"
    volumes:
      - ./fluent.conf:/fluentd/etc/fluent.conf

This keeps the logging configuration visible and version-controlled.

Conclusion

A local lab for Docker logging drivers gives you a safe environment to understand driver behavior, test configuration options, and practice recovery. By following the steps in this guide, you have configured and verified the json-file, syslog, journald, fluentd, and gelf drivers, learned how to read logs from different destinations, and prepared for common failures.

Remember these key practices:

• Always set log rotation for local drivers.
• Check whether a driver supports docker logs before relying on it for debugging.
• Verify connectivity to remote log endpoints from the container's perspective.
• Use docker inspect to confirm the effective logging driver and options.
• Keep a rollback plan: store the previous daemon configuration and container run commands.

For production, choose a logging driver that matches your central log infrastructure and retention needs. Test it with realistic log volumes and a load test. Document the failure behavior and monitoring thresholds. A good logging setup is not set-and-forget; it requires periodic review and adjustment.

Start with one low-risk change in your lab, observe the result, and then apply it to a staging environment before production. This incremental approach builds confidence and reduces the risk of log loss or service disruption.