Kubernetes ConfigMaps and Secrets: Troubleshooting with Practical Examples

ConfigMaps and Secrets are the standard way to inject configuration and sensitive values into Kubernetes Pods. This guide shows how to use them via environment variables and mounted files, how updates flow to running workloads, and how to troubleshoot safely without leaking credentials. You will learn a repeatable workflow, practical examples you can run, and fixes for common mistakes.

Workflow Overview

• Design keys and scope: choose clear key names (for example APP_MODE, FEATURE_FLAG) and keep data small and focused per workload.
• Create resources: define ConfigMaps for non‑sensitive data and Secrets for sensitive values.
• Wire into Pods: use env/valueFrom for process‑level config and volumes for file‑based config.
• Validate in running Pods: list env, read mounted files, and check events.
• Plan updates: understand when a rollout is required and when volumes refresh in place (kubelet syncs ConfigMap/Secret volumes roughly every 10 seconds).
• Trigger safe rollouts: restart Deployments or use checksum annotations to roll automatically on changes.
• Monitor and rotate: watch logs/events and rotate Secrets with minimal risk.

Practical examples

Create namespace and resources

kubectl create namespace cfgdemo
kubectl -n cfgdemo create configmap app-config \
  --from-literal=APP_MODE=dev \
  --from-literal=FEATURE_FLAG=true
kubectl -n cfgdemo create secret generic app-secret \
  --from-literal=DB_USER=demo \
  --from-literal=DB_PASSWORD='s3cr3tP@ss'

Deployment that uses both env and mounted files

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  namespace: cfgdemo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: toolbox
        image: busybox:1.36
        command: ["sh","-c","sleep 3600" ]
        envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secret
        volumeMounts:
        - name: cm-vol
          mountPath: /etc/app/config
        - name: secret-vol
          mountPath: /var/run/secrets/app
          readOnly: true
      volumes:
      - name: cm-vol
        configMap:
          name: app-config
      - name: secret-vol
        secret:
          secretName: app-secret

Apply and verify

kubectl -n cfgdemo apply -f deployment.yaml
kubectl -n cfgdemo rollout status deploy/app

# Check env without exposing secrets
kubectl -n cfgdemo exec deploy/app -- sh -c 'printenv | sort | grep -E "^(APP_MODE|FEATURE_FLAG|DB_USER)="'
# Verify secret env exists but do not print its value
kubectl -n cfgdemo exec deploy/app -- sh -c '[ -n "$DB_PASSWORD" ] && echo DB_PASSWORD is set'

# Check mounted files exist and view sizes instead of contents
kubectl -n cfgdemo exec deploy/app -- sh -c 'ls -l /etc/app/config; ls -l /var/run/secrets/app; wc -c /var/run/secrets/app/DB_PASSWORD'

Update the ConfigMap and observe behavior

# Change a value in the ConfigMap
kubectl -n cfgdemo patch configmap app-config \
  --type merge -p '{"data":{"APP_MODE":"prod"}}'

# Mounted files from the ConfigMap refresh shortly; check file content
kubectl -n cfgdemo exec deploy/app -- sh -c 'cat /etc/app/config/APP_MODE'

# Env vars do not change in existing Pods; trigger a rollout to pick up new env
kubectl -n cfgdemo rollout restart deploy/app
kubectl -n cfgdemo rollout status deploy/app
kubectl -n cfgdemo exec deploy/app -- sh -c 'echo APP_MODE=$APP_MODE'

Rotate a Secret safely

# Create a new secret value
kubectl -n cfgdemo create secret generic app-secret-v2 \
  --from-literal=DB_USER=demo \
  --from-literal=DB_PASSWORD='n3wS3cr3t' \
  --dry-run=client -o yaml | kubectl apply -f -

# Point the Deployment to the new Secret name for env vars
kubectl -n cfgdemo set env deploy/app --from=secret/app-secret-v2 --keys=DB_USER,DB_PASSWORD --overwrite

# Also update the mounted secret volume reference
kubectl -n cfgdemo patch deploy app -p '{"spec":{"template":{"spec":{"volumes":[{"name":"secret-vol","secret":{"secretName":"app-secret-v2"}},{"name":"cm-vol","configMap":{"name":"app-config"}}]}}}}'

kubectl -n cfgdemo rollout status deploy/app

Safe troubleshooting commands

• List and describe without dumping secret data:

  kubectl -n cfgdemo get configmap,secret
  kubectl -n cfgdemo describe configmap/app-config
  kubectl -n cfgdemo describe secret/app-secret

• Inspect Pod wiring:

  kubectl -n cfgdemo get pod -l app=app
  kubectl -n cfgdemo get pod -l app=app -o wide
  kubectl -n cfgdemo get pod -l app=app -o yaml | grep -A3 -E 'envFrom:|volumeMounts:|volmes:'

• Verify env and files in‑container without exposing secrets:

  kubectl -n cfgdemo exec deploy/app -- printenv APP_MODE
  kubectl -n cfgdemo exec deploy/app -- sh -c 'test -f /var/run/secrets/app/DB_PASSWORD && echo secret file present'

• Only when strictly needed, decode a single key in a controlled terminal session:

  # CAUTION: This prints secret material to your screen
  kubectl -n cfgdemo get secret app-secret -o jsonpath='{.data.DB_PASSWORD}' | base64 -d; echo

• Check events and logs for mount or permissions issues:

  kubectl -n cfgdemo get events --sort-by=.lastTimestamp | tail -n 20
  kubectl -n cfgdemo logs deploy/app

Common mistakes and fixes

• Using a ConfigMap for sensitive data
• Fix: put sensitive keys in a Secret. Keep ConfigMaps for non‑sensitive settings.
• Expecting env vars to update without a rollout
• Fix: restart the Deployment after changing referenced ConfigMaps or Secrets used as env/envFrom.
• Expecting instant file updates from volumes
• Fact: mounted ConfigMap and Secret files refresh shortly after changes (≈10 s). Apps that cache must reload on change.
• Key name typos between resource and Pod spec
• Fix: cross‑check keys with kubectl describe and printenv inside the Pod.
• Wrong namespace
• Fix: pass -n to kubectl or set a default namespace for the context.
• Secret base64 confusion when writing raw YAML
• Fix: prefer kubectl create secret generic … which handles encoding. If you must hand‑edit, use echo -n 'value' | base64 to avoid newline bugs.
• File permission surprises
• Fact: Secret files are read‑only by default. If your app needs group read, set fsGroup on the Pod or defaultMode on the volume.
• Oversized blobs in ConfigMaps/Secrets
• Fix: keep under typical size limits (1 MiB) and store large artifacts elsewhere (for example, in a filesystem or object store mounted via a volume).

Rollout behavior and updates

• env and envFrom values are loaded at container start. Pods must be restarted to see changes.
• Mounted ConfigMap and Secret volumes update shortly after the resource changes. The app may still need to reload its config.
• Deployments do not automatically roll on external resource changes. Only Pod template changes trigger a new ReplicaSet.
• To force a rollout when config changes, either:
• Manually restart: kubectl rollout restart deploy/<name>
• Use a checksum annotation pattern so any ConfigMap/Secret change modifies the Pod template and triggers a rollout.

Checksum example (Linux shells)

CM_SUM=$(kubectl -n cfgdemo get cm app-config -o json | sha256sum | cut -d' ' -f1)
SEC_SUM=$(kubectl -n cfgdemo get secret app-secret -o json | sha256sum | cut -d' ' -f1)

kubectl -n cfgdemo annotate deploy/app \
  config-hash=$CM_SUM secret-hash=$SEC_SUM --overwrite
kubectl -n cfgdemo rollout status deploy/app

macOS alternative for the checksum command

CM_SUM=$(kubectl -n cfgdemo get cm app-config -o json | shasum -a 256 | cut -d' ' -f1)
SEC_SUM=$(kubectl -n cfgdemo get secret app-secret -o json | shasum -a 256 | cut -d' ' -f1)

Local Pilot Plan

Goal: validate wiring and rollout behavior in a sandbox namespace with minimal blast radius.

Scope:

• One namespace (cfgdemo), one ConfigMap, one Secret, one Deployment with 1 replica.
• Both env and volume wiring to compare behaviors.

Steps:

1. Create resources and Deployment exactly as in the Practical examples section.
2. Verify initial wiring:

• printenv APP_MODE
• confirm secret file presence and size

1. Change a ConfigMap key; confirm:

• mounted file refreshes
• env value does not change until restart

1. Restart the Deployment; confirm updated env.
2. Rotate the Secret by switching the Secret name and rolling out.
3. Add checksum annotations and re‑apply after any config change; confirm automatic rollout.

Exit criteria:

• You can prove each of the following within minutes:
• Env requires restart to update.
• Mounted files update without a rollout.
• Rollout restart or checksum annotations propagate changes predictably.

Conclusion

ConfigMaps and Secrets let you separate configuration from code while keeping sensitive values protected. In practice, validate both env and volume wiring, know when a restart is required, and prefer safe troubleshooting commands that avoid exposing secret values. Start with a small pilot, observe how updates propagate, and then adopt a repeatable rollout method such as checksum annotations. With these patterns in place, teams can change configuration confidently and resolve issues quickly.