## Intro

Docker logging drivers control where container stdout and stderr are sent, but they are often treated as a low-priority detail in CI/CD automation. That creates a dangerous gap: a pipeline can deploy a container successfully while silently losing logs, leaking sensitive output, or filling a disk because the wrong driver is active. This guide focuses on operational safety for Docker logging driver changes inside automated pipelines. It connects deployment, verification, and rollback to concrete commands, expected output, failure signals, and recovery decisions.

The target readers are developers, DevOps consultants, and technical startup teams who already use Docker but want a controlled workflow for changing logging drivers. The core rule is: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and know how to recover if the expected state is not reached.

## Version and Environment Inventory

Before touching any logging configuration, identify the exact Docker version, the deployed topology, and the relevant container or service. This prevents copy-paste mistakes that assume a newer Docker feature or a different orchestration layer.

Start with read-only inventory commands. On a standalone Docker host, run:

```bash
docker version --format '{{.Server.Version}}'
```

Typical output: `20.10.24` or `24.0.7`. Logging driver support varies by version; for example, the `awslogs` driver was added in Docker 1.12, and `journald` behavior can differ across Docker 19.x and 20.x. Do not assume a driver exists just because it is mentioned in documentation.

Next, list running containers and their details:

```bash
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
```

For a specific container, inspect the currently active logging driver and its options:

```bash
docker inspect <container> --format '{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}'
```

Example output: `json-file map[max-size:10m max-file:3]`. This confirms the driver is `json-file` with rotation. If the output is `json-file map[]`, only default rotation applies, which can lead to unbounded log growth.

For Compose-based projects, the equivalent inventory is:

```bash
docker compose config
```

This outputs the full effective configuration, including any `logging:` section for each service. If the logging section is absent, Docker defaults to the daemon-level `json-file` driver.

To verify data persistence status before any change, inspect mounts:

```bash
docker inspect <container> --format '{{json .Mounts}}'
```

A named volume such as `app_data:/var/lib/app` is managed by Docker and survives container recreation. A bind mount such as `./data:/var/lib/app` depends on the host path existing with correct permissions. If your application writes logs to a file inside the container, that file lives in the container layer and will be lost on restart unless explicitly mounted or forwarded to stdout. Confirm where log data actually lives before altering logging configuration.

Finally, run a restart test on a staging clone:

```bash
docker stop <container> && docker start <container>
docker logs --tail 20 <container>
```

If the last 20 lines show the application starting cleanly and receiving traffic, the current logging path is at least stable enough to compare against after the change.

## Safe Configuration Path

Changing a logging driver is a configuration change, not a code change, and it deserves the same review and rollback discipline. The safest path is to define the intended logging configuration in a declarative file, review it outside production, validate it in a local or staging environment, and only then deploy it through the pipeline.

### Step 1: Pin the Docker or Compose file

For a standalone container launched directly by the pipeline, the configuration might live in a Docker run command or a systemd unit. For Compose, it lives in `docker-compose.yml` or `compose.yml`. Example service definition:

```yaml
services:
  web:
    image: nginx:1.25
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
```

This replaces the default with explicit rotation: three files of up to 10 MB each. For comparison, the default `json-file` driver with no options can grow a single log file indefinitely until disk is exhausted.

### Step 2: Choose a specific candidate driver and justify the change

For centralized logging, common choices are Fluentd (`fluentd`), AWS CloudWatch (`awslogs`), or systemd journal (`journald`). The decision must include the operational tradeoff. For example, switching from `json-file` to `fluentd` enables log shipping to an aggregator, but it removes the `docker logs` command's ability to show container output unless a fluentd log driver fallback or extra configuration is set. That loss of visibility can break debugging flows in production.

A safer incremental move is often to keep `json-file` but add rotation options. If centralized logging is required, configure the `fluentd` driver with explicit connection details and retry settings:

```yaml
logging:
  driver: fluentd
  options:
    fluentd-address: localhost:24224
    fluentd-async: "true"
    fluentd-retry-wait: "5"
    fluentd-max-retries: "10"
```

### Step 3: Validate the configuration statically

Run `docker compose config` after editing the Compose file. Example output includes a `logging:` block under the service. If the driver name is misspelled, Docker will fail at container creation with an error like:

```
failed to initialize logging driver: error looking up logging plugin fluentdd: plugin not found
```

Catching this in the pipeline before deployment is cheaper than after.

### Step 4: Test in a local environment

Spin up the service with `docker compose up -d` and check the driver takes effect:

```bash
docker inspect <container> --format '{{.HostConfig.LogConfig.Type}}'
```

Expected output: the new driver name, e.g. `fluentd`.

Then generate test log lines and confirm they appear in the destination:

```bash
docker exec <container> sh -c 'echo test-log-$(date +%s)'
```

If using Fluentd, query the aggregator for the recent line:

```bash
fluentd-cat test.tag < /dev/null && curl -s http://localhost:9880/api/plugins.json | jq '.plugins[] | select(.type=="output") | .config'
```

The exact verification depends on your logging backend, but the principle is: produce a unique event, then search for it in the target system. If the event does not appear within a few seconds, the pipeline is broken.

## Verification and Diagnostics

Verification for logging driver changes has three layers: driver acceptance, log flow, and operational failure detection.

### Driver acceptance

Immediately after a deployment, run:

```bash
docker ps --filter "name=<service>" --format '{{.Names}} {{.Status}}'
```

If the container status is not `Up`, the service is down. Then check the driver:

```bash
docker inspect <container> --format '{{.HostConfig.LogConfig.Type}}'
```

Expected: the new driver name. If the driver is unchanged, the deployment likely reused an old container or the pipeline did not apply the Compose changes. Force recreation with `docker compose up -d --force-recreate <service>` and re-check.

### Log flow verification

For `json-file`, rotation can be verified by checking the files on the host. Find the container ID:

```bash
docker inspect <container> --format '{{.Id}}'
```

Then look at the host directory (example path on Linux):

```bash
ls -lh /var/lib/docker/containers/<container-id>/<container-id>-json.log*
```

You should see multiple files if rotation is active (e.g. `-json.log`, `-json.log.1`, `-json.log.2`). If only one file exists and it has grown beyond the `max-size`, rotation is not working.

For `fluentd`, check the fluentd process or endpoint on the aggregator. A common failure is `fluentd-address` pointing to a stale host. Ping the host from the container:

```bash
docker exec <container> sh -c 'nc -zv <fluentd-host> 24224'
```

Expected output: `Connection to <fluentd-host> 24224 port [tcp/*] succeeded!` If connection refused, the driver will retry according to `fluentd-max-retries` and then drop logs. Check the container's own stderr for driver errors:

```bash
docker logs <container> 2>&1 | grep -i fluentd
```

Messages like `Failed to write to fluentd` indicate the driver is failing.

### Failure detection in the pipeline

Add a post-deployment smoke test that calls the log source. For a web service, trigger a request with a unique trace ID:

```bash
curl -H 'X-Request-ID: verify-log-123' http://localhost:8080/health
```

Then poll the logging backend for that ID. If the log is expected in a file, use `docker exec` to read the container's log file:

```bash
docker exec <container> sh -c 'grep verify-log-123 /var/log/app/*.log'
```

If nothing returns after three attempts over 30 seconds, fail the pipeline and initiate rollback.

## Failure Modes and Recovery

The most common failure modes for Docker logging driver changes are:

### 1. Container fails to start due to invalid driver or option

**Symptom**: Pipeline reports container creation error, e.g. `invalid log driver: fluent` or `invalid option: max-size`.

**Why**: Typo in Compose file or driver not installed (e.g. `awslogs` requires the Docker daemon to have the cloudwatch logs plugin).

**Recovery**: Roll back the Compose file to the previous commit. Immediate hotfix: run `docker update --log-driver json-file <container>` if the container exists, but this does not change the file, so the next recreate will break again. Fix the declaration and redeploy.

### 2. Logs stop appearing in the central system after switch

**Symptom**: In the aggregator (e.g. Kibana, CloudWatch), the service's logs stop updating, but `docker ps` shows the container running.

**Why**: The logging driver cannot connect to the remote (e.g. Fluentd host down, AWS credentials missing) or the driver buffers indefinitely due to heavy load.

**Recovery**: Temporarily switch back to `json-file` with rotation to restore `docker logs` visibility. Then debug the remote connectivity separately. Never leave production with the remote logging driver if it is not verified end-to-end.

### 3. Disk fills up after switching to `journald` or default `json-file`

**Symptom**: Server disk usage climbs, and `docker ps` shows unhealthy containers or crashes.

**Why**: `journald` driver has its own rotation settings but can still consume large space if not limited, or the daemon's `json-file` default is tiny and fills quickly.

**Recovery**: Set explicit rotation for `json-file` first. For `journald`, configure `SystemMaxUse` in `/etc/systemd/journald.conf` and restart `systemd-journald`. Monitor disk space as a pipeline metric.

### 4. Rolling update loses logs for a short window

**Symptom**: During `docker stack deploy` or `docker compose up -d` with a logging change, old container is stopped and new one started. For a few seconds, logs from the old container might not be flushed.

**Why**: Some drivers flush asynchronously. The old container can be killed before its buffered logs are written.

**Recovery**: Accept the small loss if the application is tolerant, or use a sidecar log shipper that reads from a file mount, which decouples log collection from the driver lifecycle. Avoid relying solely on the logging driver for high-value audit logs.

To prepare for recovery, always tag the previous working image or Compose revision. In a GitOps pipeline, a rollback is a `git revert` of the logging change, followed by deploying the previous commit. Keep the previous driver configuration visible in the repository history so operators can compare.

## Operations Checklist

Use this checklist before and after a logging driver change in CI/CD. Each item lists the accountable owner for a typical startup team and the review frequency.

| # | Item | Owner | How to verify | Frequency |
|---|------|-------|---------------|-----------|
| 1 | Docker version and daemon support for the candidate driver | DevOps engineer | `docker version` and `docker info --format '{{.LoggingDrivers}}'` | Before each change |
| 2 | Current driver and options for the target service | Service owner (e.g. Backend team lead) | `docker inspect` output | Before each change |
| 3 | Log destination health (e.g. Fluentd endpoint reachable, CloudWatch permissions) | Platform team lead | Run connectivity check from a test container | Before each change |
| 4 | New logging config declared in Compose or Dockerfile | Service owner | `docker compose config` diff | Before merge |
| 5 | Static validation passes | CI pipeline | `docker compose config` exit code 0 | Every pipeline run |
| 6 | Rotational limits set (max-size, max-file) | DevOps engineer | `docker inspect` shows non-empty config | Before merge |
| 7 | Staging deployment succeeds with new driver | Service owner | Container status `Up` and driver changed | After staging deploy |
| 8 | Log event generated and observed in central system | QA lead | Search for unique test string | After staging deploy |
| 9 | Rollback path defined and tested | DevOps engineer | Simulate failure and revert | After staging deploy |
| 10 | Production deployment smoke test includes log check | SRE lead | Pipeline step fails if log missing | Every production deploy |

Accountable owners are named individuals, not groups. If the team is smaller than five people, the same person may own multiple items, but each item must have a single point of contact who can say yes or no before proceeding. Revisit the driver choice quarterly: logging backends change, log volume grows, and new Docker versions add or deprecate drivers.

## Common Pitfalls and How to Avoid Them

### Pitfall 1: Assuming Docker's default logging is good enough

Many teams never configure logging and rely on the daemon default `json-file` with no rotation. On a busy container, the log file can grow to tens of gigabytes and fill the host disk.

**Why it happens**: Lack of visibility into host disk usage and a false sense that Docker handles logs automatically.

**Prevention**: Set `max-size` and `max-file` on every container, either in Compose or via daemon defaults. Monitor disk usage per container with a host-agent metric.

**Recovery**: Rapidly truncate the log file with `truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log` (requires root) or restart the container with rotation configured.

### Pitfall 2: Switching to a centralized logging driver without preserving `docker logs` for debugging

Developers often rely on `docker logs` for quick troubleshooting. When you switch to `fluentd` or `syslog`, `docker logs` may show nothing, and devs are forced to use a different tool they are unfamiliar with.

**Why it happens**: The team equates centralized logging with improved debugging without considering the local workflow.

**Prevention**: Keep `json-file` as the driver for development and staging, and use a sidecar or daemon-level configuration to ship logs. In production, document the equivalent log query for the central system.

**Recovery**: Install a local aggregator such as Fluent Bit that accepts logs from the remote driver and writes to a file for quick `grep`, or run a temporary sibling container with `docker run --log-driver=json-file` for debugging.

### Pitfall 3: Ignoring log rotation when using `journald`

The `journald` driver hands logs to journald, but journald has its own storage limits that are often left default. On a system with a small `/var/log/journal` partition, this can crash the journal and break system logging, not just container logs.

**Why it happens**: Operators only configure Docker options and forget the underlying system service.

**Prevention**: Configure `SystemMaxUse=1G` in `/etc/systemd/journald.conf` and restart journald. Monitor journal size.

**Recovery**: If the journal partition is full, run `journalctl --vacuum-size=500M` to free space immediately, then adjust the config.

### Pitfall 4: Changing logging driver in the middle of a rolling update without draining

In a rolling update, old containers are stopped as new ones come up. If the old container has pending buffered logs and the new container uses a different driver, the pending logs might be lost.

**Why it happens**: The deployment process does not account for the driver's flush behavior.

**Prevention**: Drain logs by waiting for quiet period or sending a signal that flushes before stopping. Alternatively, design logging to be directly from the application to a sidecar, bypassing the driver.

**Recovery**: In most cases, the lost logs are low-volume startup messages, but for audit logs, consider replicating to a file mount that is read by a log shipper.

## Conclusion

Docker logging drivers CI/CD automation is a repeatable process, not a one-off command. Every change must be version-scoped, observable, and reversible. The workflow described here follows a consistent pattern: inventory the current state, declare the desired state in a reviewed file, validate locally, deploy through a pipeline that verifies log flow, and maintain a rollback commit.

As a next step, pick one container in a staging environment that currently relies on default `json-file`. Add explicit rotation options in Compose, deploy, and verify that the container starts with the options and that log rotation functions after generating a large volume of test logs. Document the exact commands and expected output in your team's runbook, then schedule a quarterly review to reassess driver choices as log volume and infrastructure evolve.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Logging is the last mile of observability; treating it with the same rigor as application code keeps teams able to debug, audit, and operate with confidence.