## Intro

Docker logging drivers control how container logs are collected and stored. The default `json-file` driver writes logs locally, but other drivers can send logs to a remote system, add metadata, or reduce disk usage. Setting up a local lab lets you test driver behavior without touching production environments.

This guide walks through a complete Docker logging driver lab on a single Linux host. You will inspect the current logging setup, configure and test several drivers, learn how to rotate and read logs, and practice recovery when things go wrong. Every step includes commands you can run and the expected output, so you can verify each result before moving on.

The examples assume Docker Engine 24.0 or later, with the Docker CLI and Docker Compose v2 installed. Without systemd journal access or rsyslog, some drivers may not work as shown. We will cover the prerequisites for each driver.

## Version and Environment Inventory

Before changing any configuration, record your Docker version, storage driver, and current logging setup. This gives you a rollback point and helps you spot incompatible options.

Run these commands and save the output:

```bash
docker version --format '{{.Server.Version}}'
docker info --format 'Storage Driver: {{.Driver}}'
docker info --format 'Logging Driver: {{.LoggingDriver}}'
```

Expected output on a fresh Docker installation may look like this:

```text
24.0.7
Storage Driver: overlay2
Logging Driver: json-file
```

If your output differs, note it. For example, some cloud images set `journald` or `syslog` as the default driver. The lab assumes you can override the driver per container at runtime, which all modern Docker versions support.

Also check that you have permission to run Docker without `sudo`. If not, add your user to the `docker` group and log out and back in. This is not strictly required, but it avoids permission errors when reading log files.

## Safe Configuration Path

We will configure each logging driver with a minimal set of options and then run a container that writes predictable log lines. This keeps the changes small and easy to undo.

### Default json-file driver and log rotation

The `json-file` driver stores logs as JSON objects under `/var/lib/docker/containers/<container-id>/<container-id>-json.log`. Without rotation, this file can grow until the disk fills. Production best practice is to set `max-size` and `max-file`.

Start a container with explicit rotation settings:

```bash
docker run -d --name log-json \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  alpine sh -c 'while true; do echo "$(date) hello from json-file"; sleep 1; done'
```

This writes one line per second. To create enough data for rotation, you can temporarily lower `max-size` to `1k` in a second container:

```bash
docker run -d --name log-rotate-test \
  --log-driver json-file \
  --log-opt max-size=1k \
  --log-opt max-file=3 \
  alpine sh -c 'while true; do echo "$(date) rotate me"; done'
```

After a minute or so, list the log files:

```bash
sudo ls -lh /var/lib/docker/containers/$(docker inspect -f '{{.Id}}' log-rotate-test)/
```

You should see multiple files ending in `-json.log`, `-json.log.1`, `-json.log.2`, and possibly a compressed `.gz` file, depending on your Docker version. This confirms rotation works.

Stop and remove the test containers before continuing:

```bash
docker rm -f log-json log-rotate-test
```

### Syslog driver

The `syslog` driver sends logs to a syslog daemon. On most Linux hosts, `rsyslog` or `syslog-ng` is already running. If not, install rsyslog:

```bash
sudo apt update && sudo apt install -y rsyslog   # Debian/Ubuntu
sudo systemctl start rsyslog
```

Run a container using the syslog driver:

```bash
docker run -d --name log-syslog \
  --log-driver syslog \
  --log-opt syslog-address=udp://127.0.0.1:514 \
  --log-opt tag="docker/{{.Name}}" \
  alpine sh -c 'echo "syslog test from container" && sleep 60'
```

Check the host syslog:

```bash
sudo tail -n 5 /var/log/syslog | grep 'syslog test'
```

Expected output includes a line with the tag `docker/log-syslog` and the message. If you see nothing, confirm rsyslog is listening on UDP 514:

```bash
sudo ss -ulnp | grep 514
```

The output should show a process like `rsyslogd` listening on `0.0.0.0:514`. If not, edit `/etc/rsyslog.conf` and uncomment the lines for UDP syslog reception, then restart rsyslog.

Stop the container after verifying:

```bash
docker rm -f log-syslog
```

### Journald driver

The `journald` driver writes logs to the systemd journal. This requires the Docker daemon to run on a host with systemd. Check if journald is available:

```bash
systemctl is-active systemd-journald
```

Expected output: `active`. If it says `inactive` or `failed`, you need to start it with `sudo systemctl start systemd-journald`.

Run a container with the journald driver:

```bash
docker run -d --name log-journald \
  --log-driver journald \
  --log-opt tag="docker-journald-test" \
  alpine sh -c 'echo "journald test" && sleep 60'
```

Read the log from the journal:

```bash
journalctl CONTAINER_NAME=log-journald --no-pager
```

Expected output includes the `journald test` line along with systemd metadata. If you see a permission error, either run `journalctl` with `sudo` or ensure your user is in the `systemd-journal` group.

### Fluentd driver

The `fluentd` driver sends logs to a Fluentd instance. This is useful when you already run Fluentd for log aggregation. For a quick lab, start a Fluentd container that listens on TCP port 24224 and writes forwarded logs to stdout.

Create a minimal Fluentd config file `fluent.conf`:

```text
<source>
  @type forward
  port 24224
</source>
<match **>
  @type stdout
</match>
```

Run Fluentd:

```bash
docker run -d --name fluentd-lab \
  -p 24224:24224 \
  -v $(pwd)/fluent.conf:/fluentd/etc/fluent.conf \
  fluent/fluentd:v1.16-1
```

Wait a few seconds, then run a container that uses the Fluentd driver:

```bash
docker run --rm --name log-fluentd \
  --log-driver fluentd \
  --log-opt fluentd-address=127.0.0.1:24224 \
  alpine echo "fluentd test from container"
```

Check Fluentd's own logs to see the forwarded entry:

```bash
docker logs fluentd-lab | tail -n 3
```

Expected output includes a line containing `"fluentd test from container"` in the Fluentd event format. If the connection fails, check that Fluentd is listening:

```bash
docker logs fluentd-lab | grep 'listening'
```

You should see a line like `listening fluent socket on 0.0.0.0:24224`.

Stop Fluentd when done:

```bash
docker rm -f fluentd-lab
```

### Gelf driver (optional)

The `gelf` driver sends logs to a Graylog or other GELF-compatible endpoint. For a lab, you can use a simple UDP listener to test the format. Here is a quick Python UDP server that prints received datagrams:

```bash
docker run --rm -d --name gelf-listener \
  -p 12201:12201/udp \
  python:3.11-alpine python -c "
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('0.0.0.0', 12201))
while True:
    data, addr = s.recvfrom(65535)
    print(data.decode(), flush=True)
"
```

Now run a container that logs to GELF UDP:

```bash
docker run --rm --name log-gelf \
  --log-driver gelf \
  --log-opt gelf-address=udp://127.0.0.1:12201 \
  alpine echo "gelf test"
```

Check the listener output:

```bash
docker logs gelf-listener
```

You should see a GELF JSON payload containing the `gelf test` message. If not, verify the UDP port mapping with `docker port gelf-listener`.

Clean up:

```bash
docker rm -f gelf-listener
```

## Verification and Diagnostics

After configuring a driver, you must confirm that logs are actually flowing and that the driver is not silently dropping messages.

### Verify driver selection

Use `docker inspect` to see which logging driver a container uses and its options:

```bash
docker inspect -f '{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}' <container-name>
```

For example, for the `log-json` container you ran earlier, the output would be:

```text
json-file map[max-file:3 max-size:10m]
```

Run this for each container after creation to catch typos in driver names or options.

### Read logs with docker logs

`docker logs` works for most drivers, but it may behave differently. The `json-file` and `journald` drivers support `docker logs` natively. The `syslog` and `fluentd` drivers do not return logs via `docker logs`; you must query the external destination.

Test this behavior:

```bash
docker logs log-syslog
```

Expected: an error like `the logging driver "syslog" does not support the "logs" subcommand` or no output at all. This is a known limitation. Plan your diagnostics accordingly: for remote drivers, check the receiving endpoint instead of `docker logs`.

### Inspect log files on disk

For the `json-file` driver, you can directly read the log file to see the JSON structure:

```bash
sudo cat /var/lib/docker/containers/$(docker inspect -f '{{.Id}}' <container>) /<container>-json.log | head -n 2
```

Each line is a JSON object with keys `log`, `stream`, and `time`. For example:

```json
{"log":"hello from json-file\n","stream":"stdout","time":"2025-03-20T15:04:05Z"}
```

This is useful for scripting or when debugging log parsing tools.

### Check driver metrics

Docker exposes logging driver errors in the daemon logs. To see if the daemon is having trouble with a driver, check the Docker daemon journal:

```bash
sudo journalctl -u docker.service --since "10 minutes ago" | grep -i 'log'
```

Look for messages like `Failed to log message` or driver-specific errors. For the syslog driver, a common error is `connection refused` if the syslog address is wrong.

## Failure Modes and Recovery

Logging drivers can fail in ways that are not immediately obvious. Knowing the common failure modes helps you recover quickly and design robust logging setups.

### Failure: json-file logs fill the disk

Symptom: Containers stop writing logs, or the Docker daemon reports `no space left on device`. The root cause is usually no log rotation.

Recovery:

1. Find the largest log files:

   ```bash
   sudo find /var/lib/docker/containers -name '*-json.log' -size +100M -ls
   ```

2. Truncate oversized logs immediately to free space:

   ```bash
   sudo truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log
   ```

3. Prevent recurrence by setting `max-size` and `max-file` on the container. You can also set global defaults in the daemon configuration file `/etc/docker/daemon.json`:

   ```json
   {
     "log-driver": "json-file",
     "log-opts": {
       "max-size": "10m",
       "max-file": "3"
     }
   }
   ```

   Restart Docker after changing `daemon.json`:

   ```bash
   sudo systemctl restart docker
   ```

   Note: This restart will stop all running containers. Do this during a maintenance window.

### Failure: Syslog driver cannot connect

Symptom: Container starts, but no logs appear in `/var/log/syslog`. The Docker daemon log shows `connection refused` for the syslog address.

Recovery:

1. Verify the syslog daemon is running and listening on the expected socket:

   ```bash
   sudo ss -ulnp | grep 514
   ```

2. If nothing is listening, start rsyslog:

   ```bash
   sudo systemctl start rsyslog
   ```

3. If running a containerized syslog server, ensure its port is published correctly. For the earlier example, the syslog receiver used `udp://127.0.0.1:514`. If you run the receiver in a container, you need to publish UDP port 514 and use the host IP.

4. To avoid future connection failures, add a retry and a longer timeout to the driver options:

   ```bash
   --log-opt syslog-address=udp://127.0.0.1:514 \
   --log-opt syslog-retries=5 \
   --log-opt syslog-timeout=10s
   ```

### Failure: Journald driver drops logs on restart

Symptom: Logs from a container are missing after the container restarts. The `journald` driver tags logs with the container ID, not the name. If the container is removed and recreated, the ID changes, and older logs are harder to find.

Recovery:

1. Use the container ID to query logs:

   ```bash
   journalctl CONTAINER_ID=<full-container-id>
   ```

2. Add a custom tag that includes the container name for easier filtering:

   ```bash
   --log-opt tag="myapp-{{.Name}}"
   ```

   Then query with:

   ```bash
   journalctl SYSLOG_IDENTIFIER=myapp-<name>
   ```

3. In Compose files, you can set a stable container name with `container_name` to make the tag predictable.

### Failure: Fluentd driver buffers fill up

Symptom: Container logs are delayed, or the Fluentd driver reports buffer overflow. The Fluentd driver buffers logs in memory when the Fluentd endpoint is slow or unreachable. If the buffer fills, the container may block or drop logs.

Recovery:

1. Check the Docker daemon log for Fluentd errors:

   ```bash
   sudo journalctl -u docker.service | grep fluentd
   ```

2. Increase the Fluentd buffer limit if you expect bursts:

   ```bash
   --log-opt fluentd-buffer-limit=64m
   ```

3. Ensure the Fluentd endpoint has enough capacity. Tune Fluentd's flush intervals and buffer sizes on the server side.

4. For production, use a dedicated log collector with proper queueing, such as Fluentd with a file buffer.

### Failure: Gelf driver loses logs over UDP

Symptom: Some logs never appear at the GELF endpoint. UDP is connectionless and does not guarantee delivery. Under load or network issues, packets can be lost.

Recovery:

1. Use TCP instead of UDP for GELF if your endpoint supports it:

   ```bash
   --log-opt gelf-address=tcp://127.0.0.1:12201
   ```

2. Increase the `gelf-compression-level` to reduce bandwidth, but this does not prevent UDP loss. For reliable delivery, use a driver that supports TCP or a message queue.

3. Monitor the endpoint for missing sequence numbers or gaps in logs.

## Common Pitfalls and How to Avoid Them

Several mistakes recur when working with Docker logging drivers in a lab or production. Being aware of them saves time and prevents data loss.

### Ignoring driver support for docker logs

Many users assume `docker logs` works for all drivers. It does not. The `json-file`, `journald`, and `local` drivers support `docker logs`. The `syslog`, `fluentd`, `gelf`, and `awslogs` drivers do not. If you rely on `docker logs` for debugging, choose a compatible driver.

**How to avoid:** Before selecting a driver, check the Docker documentation for its `docker logs` support. Test with a small container early.

### Not setting log rotation for json-file

The default `json-file` driver has no size limit. Over months, logs can consume all disk space, causing Docker and other services to fail.

**How to avoid:** Always set `max-size` and `max-file` on every container using `json-file`. Better yet, set global defaults in the daemon configuration.

### Using bind mounts for log files without proper permissions

When mounting a directory from the host to store log files, the container user may not have write permission, causing the application to fail to start.

**How to avoid:** Check the container's user and the mount permissions. Use a named volume instead:

```bash
docker volume create app-logs
docker run -d --name app \
  -v app-logs:/var/log/app \
  --log-driver json-file \
  myapp
```

Named volumes are managed by Docker and handle permissions more gracefully.

### Mixing logging driver options with daemon-level settings

If you set a global logging driver in `daemon.json` but also pass `--log-driver` on the container, the container option overrides the global. This can lead to inconsistent logging.

**How to avoid:** Document the intended driver per service. Use explicit container-level options for special cases only. Verify with `docker inspect` after startup.

### Forgetting to publish UDP ports for remote logging receivers

When running a syslog or GELF receiver in a container, you must publish its UDP port. Omitting `-p 514:514/udp` results in silent failure; the sending container cannot reach the receiver on localhost.

**How to avoid:** Always check port mappings with `docker port <receiver-container>`. Test connectivity with `nc -u <host> <port>` before relying on the setup.

### Not cleaning up lab containers and files

After testing, orphan containers and leftover config files can interfere with later experiments or consume resources.

**How to avoid:** Keep a cleanup script. For example:

```bash
docker rm -f log-json log-syslog log-journald log-fluentd log-gelf
rm -f fluent.conf
```

Regularly run `docker system prune` in a lab environment, but be careful in shared or production hosts.

## Operations Checklist

Use this checklist whenever you deploy or modify a Docker logging driver in a lab or production-like environment.

| Step | Action | Owner | Revisit Frequency |
|------|--------|-------|-------------------|
| 1 | Confirm Docker version and daemon logging defaults with `docker info` | Platform Engineer | Monthly |
| 2 | Document the intended log destination and retention policy per service | DevOps Lead | Quarterly |
| 3 | Apply log rotation for json-file driver globally in `daemon.json` | Platform Engineer | On change |
| 4 | Test each driver with a throwaway container and verify logs at the destination | Developer | Per deployment |
| 5 | Check disk usage and log sizes weekly | Monitoring Team | Weekly |
| 6 | Review driver compatibility with `docker logs` for debugging needs | Developer | Per service |
| 7 | Run a failure drill: stop the log receiver and observe driver behavior | SRE | Quarterly |
| 8 | Update documentation after any driver change | Technical Writer | On change |

Each item has a single owner who is accountable for execution and follow-up. The revisit frequency ensures the checklist remains relevant as infrastructure evolves.

### Configuration example using Docker Compose

A complete Compose file for a web service with json-file logging and rotation:

```yaml
version: '3.8'
services:
  web:
    image: nginx:1.25
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    deploy:
      resources:
        limits:
          memory: 128M
```

For a service that sends logs to a Fluentd sidecar, you can define:

```yaml
services:
  app:
    image: myapp:latest
    logging:
      driver: fluentd
      options:
        fluentd-address: "fluentd:24224"
        tag: "app.{{.Name}}"
  fluentd:
    image: fluent/fluentd:v1.16-1
    ports:
      - "24224:24224"
    volumes:
      - ./fluent.conf:/fluentd/etc/fluent.conf
```

This keeps the logging configuration visible and version-controlled.

## Conclusion

A local lab for Docker logging drivers gives you a safe environment to understand driver behavior, test configuration options, and practice recovery. By following the steps in this guide, you have configured and verified the `json-file`, `syslog`, `journald`, `fluentd`, and `gelf` drivers, learned how to read logs from different destinations, and prepared for common failures.

Remember these key practices:

- Always set log rotation for local drivers.
- Check whether a driver supports `docker logs` before relying on it for debugging.
- Verify connectivity to remote log endpoints from the container's perspective.
- Use `docker inspect` to confirm the effective logging driver and options.
- Keep a rollback plan: store the previous daemon configuration and container run commands.

For production, choose a logging driver that matches your central log infrastructure and retention needs. Test it with realistic log volumes and a load test. Document the failure behavior and monitoring thresholds. A good logging setup is not set-and-forget; it requires periodic review and adjustment.

Start with one low-risk change in your lab, observe the result, and then apply it to a staging environment before production. This incremental approach builds confidence and reduces the risk of log loss or service disruption.