Kubernetes Logging Architecture Explained with Practical Examples

Intro

Kubernetes logging is a foundational operational capability that enables developers and DevOps engineers to understand application behavior, diagnose failures, and monitor cluster health. However, many teams struggle with fragmented logs, no clear architecture, and ad-hoc debugging. This article provides a practical, example-driven guide to Kubernetes logging architecture. You will learn how logs flow from containers to a centralized backend, how to configure common logging agents, and how to troubleshoot typical issues using concrete commands and expected outputs.

We focus on real-world scenarios: a simple single-node cluster for learning, a production multi-node cluster, and the steps to evolve from basic logging to a robust centralized setup. By the end, you will be able to design, implement, and verify a logging solution that meets operational needs.

Version and Environment Inventory

Before diving into logging, you must know your environment. This section provides commands to inventory versions, nodes, and existing logging components. Accurate inventory prevents mismatched configurations and helps you identify what is already in place.

Kubernetes and Node Information

Start by confirming the Kubernetes version and node details. This is essential because logging agent versions often have compatibility constraints with the Kubernetes API and container runtime.

kubectl version --short
# Expected output similar to:
# Client Version: v1.25.3
# Server Version: v1.25.3

kubectl get nodes -o wide
# Example output:
# NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
# node1    Ready    control-plane   10d   v1.25.3   192.168.1.10   <none>        Ubuntu 22.04.1 LTS   5.15.0-56-generic   containerd://1.6.8

Existing Logging Components

Check if any logging agents are already deployed. Common agents include Fluent Bit, Fluentd, Filebeat, or Vector. Look for DaemonSets (one per node) or Deployments.

kubectl get daemonsets --all-namespaces | grep -E 'fluent|filebeat|log|vector'
kubectl get deployments --all-namespaces | grep -E 'fluent|filebeat|log|vector'
# If nothing returns, no common agent is deployed.

# Also check for logging-related custom resources
kubectl get crd | grep -E 'fluent|log|elastic'

Container Runtime Log Location

Understanding where container logs are stored on the node is crucial for agent configuration. For containerd (the default in many clusters), logs are typically under /var/log/containers (symlinks) and /var/log/pods. For Docker, they are under /var/lib/docker/containers.

# On a node, list sample log files
ls -l /var/log/containers/ | head -5
# Example: total 0
# lrwxrwxrwx 1 root root 107 Jan  5 10:00 app-nginx-6d4b7c9f5-abcde_default_nginx-abc123.log -> /var/log/pods/default_app-nginx-6d4b7c9f5-abcde/nginx/0.log

Practical Verification

Run a test pod that produces logs and verify that they are written to the expected location.

kubectl run logtest --image=busybox --restart=Never -- sh -c 'while true; do echo "Hello from logtest at $(date)"; sleep 5; done'
kubectl logs -f logtest
# You should see repeated output.

# On the node, find the log file for this pod
ls /var/log/pods/default_logtest/ | grep logtest
# Example: logtest.log
cat /var/log/pods/default_logtest/logtest/0.log
# Shows the same output as kubectl logs.

Clean up the test pod:

kubectl delete pod logtest

Safe Configuration Path

Configuring logging agents can impact all applications on a node if done incorrectly. This section outlines a safe, incremental path: from a single-node test to a full rollout, with verification at each step.

Choosing a Logging Agent

Fluent Bit is a popular lightweight choice for Kubernetes. It is efficient and integrates well with many backends. Here we will use Fluent Bit as an example, but the principles apply to other agents.

Step 1: Deploy Fluent Bit as a DaemonSet in a Test Namespace

Start with a minimal configuration that reads container logs and outputs to stdout (for verification). Create a file fluent-bit-test.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: logging-test
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging-test
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Daemon       off
        Log_Level    info
    [INPUT]
        Name         tail
        Path         /var/log/containers/*.log
        Parser       docker
        Tag          kube.*
    [OUTPUT]
        Name         stdout
        Match        *
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging-test
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:2.1.10
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config
          mountPath: /fluent-bit/etc/
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config
        configMap:
          name: fluent-bit-config

Apply and verify:

kubectl apply -f fluent-bit-test.yaml
kubectl get pods -n logging-test -o wide
# Expect one fluent-bit pod per node, status Running.

# Check logs to see if it is collecting
kubectl logs -n logging-test daemonset/fluent-bit --tail=20
# You should see log entries from other pods if they exist, or at least the agent's own startup logs.

Step 2: Add Kubernetes Metadata Filter

Raw container logs lack pod names, namespaces, and labels. Add a Kubernetes filter to enrich logs. Update the ConfigMap fluent-bit.conf to include:

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Kube_Tag_Prefix     kube.var.log.containers.
    Merge_Log           On
    Merge_Log_Key       log_processed

This filter requires the pod to have access to the Kubernetes API. The default service account in the namespace may need RBAC permissions. For test purposes, you can create a ClusterRole and ClusterRoleBinding to allow reading pods. Create rbac.yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: default
  namespace: logging-test

Apply and restart Fluent Bit:

kubectl apply -f rbac.yaml
kubectl rollout restart daemonset/fluent-bit -n logging-test
kubectl logs -n logging-test daemonset/fluent-bit --tail=10
# Look for lines containing "kubernetes": {"pod_name": ...}

Step 3: Send Logs to a Central Backend (Elasticsearch Example)

Replace the stdout output with Elasticsearch. Ensure you have an Elasticsearch instance accessible from the cluster. For testing, you can deploy a single-node Elasticsearch in the same namespace or use an external endpoint. Update the output section in fluent-bit.conf:

[OUTPUT]
    Name            es
    Match           *
    Host            elasticsearch.logging-test.svc
    Port            9200
    Index           kubernetes-logs
    Type            _doc
    Logstash_Format On
    Logstash_Prefix kubernetes-logs
    Retry_Limit     False

Apply the config update and restart:

kubectl apply -f fluent-bit-test.yaml
kubectl rollout restart daemonset/fluent-bit -n logging-test
kubectl logs -n logging-test daemonset/fluent-bit --tail=10
# You should see no errors and possibly messages about connecting to Elasticsearch.

Verify data arrives in Elasticsearch:

# If you have curl access to Elasticsearch
curl -X GET 'http://elasticsearch.logging-test.svc:9200/_cat/indices/kubernetes-logs*?v'
# Expected output like:
# health status index                             uuid                   pri rep docs.count docs.deleted store.size pri.store.size
# green  open   kubernetes-logs-2024.01.05        aBcDeFgHiJkLmNoPqRs   1   0        100            0      200kb          200kb

Step 4: Move to Production Namespace and Scale

Once the test configuration works, replicate it in a production namespace with appropriate resource limits, node selectors, and security contexts. Consider using the official Helm chart for easier management.

helm repo add fluent https://fluent.github.io/helm-charts
helm install fluent-bit fluent/fluent-bit --namespace logging --create-namespace

Always verify after each change:

kubectl get pods -n logging -o wide
kubectl logs -n logging daemonset/fluent-bit --tail=20

Verification and Diagnostics

Once logging is configured, you need to verify that it works correctly and diagnose issues when logs are missing or malformed. This section provides diagnostic commands and interpretation.

Verify Log Collection for a Specific Pod

Create a test pod with a known log pattern:

kubectl run log-check --image=busybox --restart=Never -- sh -c 'for i in $(seq 1 10); do echo "UNIQUE_MARKER_$i"; sleep 1; done'

Wait for completion, then check if the logs appear in Fluent Bit's output. If using stdout output, check Fluent Bit logs:

kubectl logs -n logging-test daemonset/fluent-bit --since=5m | grep UNIQUE_MARKER
# You should see all 10 lines.

If using Elasticsearch, query for the marker:

curl -X GET 'http://elasticsearch.logging-test.svc:9200/kubernetes-logs*/_search?q=log:UNIQUE_MARKER_1&pretty'

Common Failure Scenarios and Diagnostics

1. No logs from any pod

• Check if Fluent Bit pods are running: kubectl get pods -n logging
• Check Fluent Bit logs: kubectl logs -n logging daemonset/fluent-bit --tail=50
• Look for errors like "permission denied" on log files. Ensure the DaemonSet has hostPath volumes mounted correctly and the container runs as root (or has appropriate permissions).

1. Logs missing for specific pods

• The tail input plugin may not have started reading the file if it existed before. Ensure Fluent Bit has read access and that the file path pattern matches. Check the Path in the input configuration. Commonly, the path is /var/log/containers/*.log.
• Verify the pod's log file exists on the node: ls -l /var/log/containers/ | grep <pod-name>

1. Logs are truncated or garbled

• The parser may be misconfigured. If using Docker parser, ensure it matches the container runtime. For containerd, you may need a different parser or no parser.
• Increase Buffer_Size and Mem_Buf_Limit in the input plugin to handle large log lines.

1. Fluent Bit cannot connect to backend

• Check network policies and service DNS. From within the cluster, test connectivity: kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- curl -v http://elasticsearch.logging-test.svc:9200
• Ensure the backend service is in the same namespace or accessible via FQDN.

Monitoring Fluent Bit Metrics

Fluent Bit exposes internal metrics that can help diagnose performance issues. Enable the metrics endpoint in the configuration:

[SERVICE]
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    2020

Then expose the port in the DaemonSet and query metrics:

kubectl port-forward -n logging daemonset/fluent-bit 2020:2020
curl http://localhost:2020/api/v1/metrics/prometheus
# Look for metrics like fluentbit_input_records_total, fluentbit_output_errors_total

Failure Modes and Recovery

Even with a well-configured logging pipeline, failures occur. This section describes common failure modes, their symptoms, and recovery steps.

Failure Mode 1: Disk Pressure from Large Logs

Symptom: Node disk usage grows rapidly, possibly causing pod evictions.

Diagnosis:

df -h /var/log
# Check disk usage on the node where logs are stored.
du -sh /var/log/containers/* | sort -rh | head -10
# Identify which pod is generating the most logs.

Recovery:

• Implement log rotation at the container runtime level (usually configured via kubelet parameters like --container-log-max-size and --container-log-max-files). For example, set in kubelet config: containerLogMaxSize: "10Mi", containerLogMaxFiles: 5.
• Add resource limits to the logging agent to prevent it from consuming too much CPU/memory.
• Implement log rate limiting in Fluent Bit using the Throttle filter.

Failure Mode 2: Logging Agent CrashLoopBackOff

Symptom: kubectl get pods -n logging shows Fluent Bit pods in CrashLoopBackOff.

Diagnosis:

kubectl describe pod -n logging -l app=fluent-bit
# Look at Events for error messages (e.g., OOMKilled, config error).
kubectl logs -n logging <fluent-bit-pod> --previous
# Check previous container logs for the crash reason.

Common causes and recovery:

• Misconfiguration: Fix the config file syntax or invalid parameters. Validate the config with fluent-bit -c /path/to/conf --dry-run locally.
• Insufficient memory: Increase memory limit in the DaemonSet spec or reduce buffering.
• Permission issues: Ensure the container runs with appropriate security context to read host logs. For example, set runAsUser: 0 if necessary, but prefer using a dedicated user with group access to the log directories.

Failure Mode 3: Backend Unavailable (Data Loss)

Symptom: Logs are not appearing in the backend, and Fluent Bit logs show repeated connection errors or retries exhausted.

Diagnosis:

kubectl logs -n logging daemonset/fluent-bit --tail=50 | grep -i 'error\|retry\|failed'
# Look for messages like "[error] [output:es:es.0] connection refused"

Recovery:

• Bring the backend service back online (e.g., Elasticsearch).
• Ensure Fluent Bit has retry logic enabled and a sufficient retry limit. In the output plugin, set Retry_Limit to a high value or False for infinite retries (with caution, as it may accumulate disk buffer).
• Consider using a disk buffer to persist logs during backend outages. In Fluent Bit, configure storage.path and use storage.type filesystem in the output.

Failure Mode 4: Log Loss Due to Pod Deletion

Symptom: When a pod is deleted, its logs are removed from the node (with the default setup). If the logging agent has not processed them yet, logs are lost.

Prevention:

• Ensure the logging agent processes logs quickly and has a large enough buffer. Monitor the fluentbit_output_retries_failed_total metric.
• Use a persistent storage for logs on the node, but this is uncommon. Better to stream logs as they are produced.
• For critical applications, consider application-level logging to a remote sink directly to avoid dependence on node log files.

Operations Checklist

This checklist summarizes essential daily and periodic operations for maintaining a healthy Kubernetes logging setup.

Daily Checks

• [ ] Verify logging agent pods are Running: kubectl get pods -n logging -o wide (expected: all Ready 1/1).
• [ ] Check for error logs from the agent: kubectl logs -n logging daemonset/fluent-bit --tail=20 | grep -i error (expected: no errors).
• [ ] Confirm logs are flowing to backend if applicable: query the backend for recent documents (e.g., Elasticsearch: curl -s 'http://elasticsearch:9200/kubernetes-logs-*/_count?pretty' | grep count).

Weekly Checks

• [ ] Inspect disk usage on nodes: df -h /var/log across nodes; ensure usage below 70%.
• [ ] Review logging agent resource usage: kubectl top pods -n logging (requires metrics server).
• [ ] Check for configuration drift: compare running ConfigMap with version-controlled desired state.

Monthly Checks

• [ ] Apply security updates to the logging agent image: check for new versions and update Helm chart or image tag.
• [ ] Test log rotation and retention: simulate a high log volume pod and ensure rotation works and old logs are removed.
• [ ] Review RBAC permissions for logging service accounts; ensure least privilege.

Incident Response Runbook

1. Logging stopped entirely

• Check agent pods: kubectl get pods -n logging
• If CrashLoopBackOff, inspect logs and events; refer to Failure Mode 2.
• If no pods, check DaemonSet/Deployment: kubectl get daemonset -n logging
• Ensure node selector or tolerations do not exclude all nodes.

1. Backend issues (Elasticsearch down)

• Check Elasticsearch pods: kubectl get pods -n elastic
• Check storage: kubectl get pvc -n elastic
• Restart if necessary, then verify Fluent Bit reconnects.

1. Log search latency high

• Check backend resource usage: kubectl top pods -n elastic
• Scale up Elasticsearch data nodes or increase heap.
• Optimize index settings (e.g., number of shards).

Conclusion

Effective Kubernetes logging requires understanding the architecture, careful configuration, and proactive monitoring. We covered the entire lifecycle: from environment inventory to safe configuration, verification, failure recovery, and ongoing operations. By following the practical examples and commands provided, you can build a resilient logging pipeline that helps you troubleshoot applications faster and maintain cluster health.

Remember that logging is not a set-and-forget system. Regularly review your configuration, monitor the health of all components, and adjust as your cluster scales. Start with the checklist provided, and adapt it to your specific environment. With these tools and practices, you will be well-equipped to handle the log management challenges in Kubernetes.

As a next step, choose one low-risk component (like enabling Kubernetes metadata filter) and test it in a non-production namespace. Document the change, verify the output, and then plan a gradual rollout to production. By taking incremental steps and verifying each one, you ensure that your logging remains a reliable source of truth for your applications.