Kubernetes Secrets Troubleshooting with Practical Examples

Kubernetes Secrets store sensitive data such as API keys, database passwords, TLS certificates, and service account tokens. When applications fail to start, crash with permission errors, or behave inconsistently across environments, the root cause is often a misconfigured or missing Secret. This guide walks through a systematic troubleshooting workflow: inventory your environment, validate Secret configuration, verify runtime consumption, diagnose common failure modes, and apply safe recovery steps. Each section includes concrete commands, expected output patterns, and decision points so you can move from symptom to resolution without guesswork.

Environment Inventory and Version Baseline

Before changing anything, capture the current state of the cluster, the workload, and the Secret itself. Start with the Kubernetes version and the Secrets backend, because encryption-at-rest configuration, CSI driver availability, and external secrets operator versions all affect behavior.

kubectl version --short
# Client Version: v1.28.4
# Kustomize Version: v5.0.4
# Server Version: v1.27.9

Check whether the cluster uses the default kube-apiserver encryption provider or an external KMS plugin:

kubectl get cm -n kube-system kube-apiserver -o yaml 2>/dev/null | grep -A5 "encryption-provider-config" || echo "No encryption config found (default: identity)"

List all Secrets in the target namespace and note their types, ages, and labels:

kubectl get secrets -n production -o custom-columns=NAME:.metadata.name,TYPE:.type,AGE:.metadata.creationTimestamp,LABELS:.metadata.labels

Typical output:

NAME                  TYPE                       AGE                   LABELS
tls-ingress-cert      kubernetes.io/tls          45d                   app=ingress,env=prod
db-credentials        Opaque                     12d                   app=api,component=postgres
docker-registry-key   kubernetes.io/dockerconfigjson 30d              app=worker,registry=ghcr
api-tokens            Opaque                     3d                    app=api,rotate=weekly

Identify the workload that consumes the Secret. For a Deployment named api-server:

kubectl get deployment api-server -n production -o yaml | grep -A10 "secretRef\|envFrom\|volumeMounts"

Record the exact key names the container expects. A mismatch between the Secret key (DB_PASSWORD) and the env reference (DATABASE_PASSWORD) is a frequent cause of CrashLoopBackOff with no obvious error in the Secret itself.

Safe Configuration Path: Creating and Updating Secrets

Treat Secret manifests as code: version-controlled, reviewed, and applied through a pipeline. Never edit live Secrets with kubectl edit in production unless you have a rollback plan.

Declarative Creation with kubectl create secret (Dry-Run First)

Generate a manifest without applying it, inspect the base64 output, then commit the YAML:

kubectl create secret generic db-credentials \
  --namespace=production \
  --from-literal=DB_HOST=postgres-primary.production.svc.cluster.local \
  --from-literal=DB_PORT=5432 \
  --from-literal=DB_USER=app_user \
  --from-literal=DB_PASSWORD='S3cur3P@ssw0rd!' \
  --dry-run=client -o yaml > db-credentials.yaml

The generated file contains base64-encoded values. Verify the encoding round-trip:

grep "DB_PASSWORD:" db-credentials.yaml | awk '{print $2}' | base64 -d
# Output: S3cur3P@ssw0rd!

Apply with server-side apply to preserve field ownership:

kubectl apply -f db-credentials.yaml --server-side --force-conflicts

Rotating Values Without Downtime

When rotating a password, create a new Secret version and update the Deployment to reference it, then roll out:

kubectl create secret generic db-credentials-v2 \
  --namespace=production \
  --from-literal=DB_PASSWORD='N3wP@ssw0rd!' \
  --from-literal=DB_HOST=postgres-primary.production.svc.cluster.local \
  --from-literal=DB_PORT=5432 \
  --from-literal=DB_USER=app_user \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl set env deployment/api-server -n production DB_PASSWORD_=$(kubectl get secret db-credentials-v2 -n production -o jsonpath='{.data.DB_PASSWORD}' | base64 -d)
# Or better: use envFrom with secretRef.name=db-credentials-v2 and rollout
kubectl patch deployment api-server -n production -p '{"spec":{"template":{"spec":{"containers":[{"name":"api","envFrom":[{"secretRef":{"name":"db-credentials-v2"}}]}]}}}}'
kubectl rollout status deployment/api-server -n production --timeout=3m

Verify the new Secret is mounted:

kubectl exec -n production deploy/api-server -- cat /etc/secrets/db-credentials/DB_PASSWORD 2>/dev/null || echo "Volume not mounted; check volumeMounts"

External Secrets Operator Pattern

If you use the External Secrets Operator (ESO) with AWS Secrets Manager, Vault, or Azure Key Vault, the troubleshooting surface shifts to the ExternalSecret and SecretStore resources:

kubectl get externalsecret -n production
kubectl describe externalsecret db-credentials -n production
# Look for Condition: Ready=True, last sync timestamp, and any "SecretSynced" events

A common ESO failure: the SecretStore references a role ARN that lacks secretsmanager:GetSecretValue permission. The event log will show Failed to fetch secret: AccessDenied.

Verification and Diagnostics at Runtime

Once the Secret exists, confirm the workload actually receives the correct values at runtime.

Pod-Level Inspection

Check the environment variables inside a running container:

POD=$(kubectl get pods -n production -l app=api -o jsonpath='{.items[0].metadata.name}')
kubectl exec -n production $POD -- env | grep -E 'DB_|PASSWORD|TOKEN'

Expected output:

DB_HOST=postgres-primary.production.svc.cluster.local
DB_PORT=5432
DB_USER=app_user
DB_PASSWORD=S3cur3P@ssw0rd!

If the variable is missing or empty, inspect the pod spec for the volume mount path:

kubectl get pod $POD -n production -o yaml | grep -A5 "volumeMounts\|secret:"

For file-based mounts (common with TLS certs), read the file directly:

kubectl exec -n production $POD -- cat /etc/tls/tls.crt | head -2
# Should show: -----BEGIN CERTIFICATE-----

Kubelet and API Server Logs

If the pod is stuck in ContainerCreating, the kubelet may be unable to mount the Secret. On a node (or via kubectl debug node/...):

journalctl -u kubelet -f | grep -i "secret\|mount"
# Look for: "Failed to mount secret" or "secret not found"

At the API server level, watch for Secret watch events that indicate propagation delays:

kubectl get events -n production --field-selector involvedObject.kind=Secret --sort-by='.lastTimestamp'

Network Policy and RBAC Side Effects

A Secret may exist and be mounted, but the application fails to use it because of downstream connectivity (e.g., cannot reach the database host) or insufficient RBAC to read the Secret in the first place. Verify the ServiceAccount has get and list on secrets in the namespace:

kubectl auth can-i get secrets -n production --as=system:serviceaccount:production:api-server
# yes

Failure Modes and Recovery Procedures

1. Secret Not Found / Mount Failure

Symptom: Pod stuck in ContainerCreating; events show MountVolume.SetUp failed for volume "secret-volume" : secret "db-credentials" not found.

Diagnosis:

kubectl get secret db-credentials -n production
# Error from server (NotFound): secrets "db-credentials" not found

Recovery:

• Recreate the Secret from backup or source of truth (Vault, SealedSecret, GitOps repo).
• If using SealedSecrets, decrypt the sealed secret:

  kubeseal --fetch-cert --controller-name=sealed-secrets -n kube-system > pub.pem
  kubeseal --cert pub.pem --format yaml < db-credentials.yaml > db-credentials-sealed.yaml
  kubectl apply -f db-credentials-sealed.yaml

• Confirm the pod recovers:

  kubectl delete pod -n production -l app=api  # Forces recreation with new Secret
  kubectl rollout status deployment/api-server -n production

2. Key Mismatch Between Secret and Container Expectation

Symptom: Application logs KeyError: 'DATABASE_PASSWORD' or starts with empty env var.

Diagnosis:

kubectl get secret db-credentials -n production -o jsonpath='{.data}' | jq -r 'keys[]'
# DB_HOST
# DB_PORT
# DB_USER
# DB_PASS   <-- Note: key is DB_PASS, not DB_PASSWORD

Recovery:

• Option A: Update the Secret to include the expected key (preferred for backward compatibility):

  kubectl patch secret db-credentials -n production -p '{"stringData":{"DB_PASSWORD":"S3cur3P@ssw0rd!"}}'

• Option B: Update the Deployment env reference to match the existing key and roll out.

3. Stale Cached Credentials After Rotation

Symptom: Application continues using old password after Secret update; database logs show authentication failures for old user.

Root Cause: The pod mounts the Secret as a volume but the application reads the file only at startup. Kubernetes updates the file in-place (atomic write via symlink swap), but the application holds an open file descriptor or caches the value.

Recovery:

• Restart the pod to force re-read:

  kubectl rollout restart deployment/api-server -n production

• For zero-downtime rotation, implement a SIGHUP reload handler in the application or use a sidecar (e.g., stakater/reloader) that watches Secret changes and triggers rolling restarts.

4. TLS Certificate Expiry or Mismatch

Symptom: Ingress controller logs x509: certificate signed by unknown authority or clients report CERT_DATE_INVALID.

Diagnosis:

kubectl get secret tls-ingress-cert -n production -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -dates -subject -issuer
# notBefore=Nov 10 00:00:00 2023 GMT
# notAfter=Nov 10 00:00:00 2024 GMT
# subject=CN = api.example.com
# issuer=CN = Let's Encrypt Authority X3

Recovery:

• If using cert-manager, trigger renewal:

  kubectl annotate certificate tls-ingress-cert -n production cert-manager.io/force-renewal="$(date +%s)" --overwrite

• For manual certs, generate a new Secret with the renewed cert/key pair and update the Ingress spec.tls[0].secretName.

5. Encryption-at-Rest Misconfiguration

Symptom: After upgrading control plane, Secrets become unreadable; kubectl get secret returns garbled data or Error from server (InternalError): Internal error occurred: failed to decrypt.

Diagnosis: The encryption-provider-config on the API server points to a KMS key that was deleted or rotated without updating the config.

Recovery:

• Restore the KMS key or update the encryption config to use a valid key, then restart kube-apiserver (rolling restart on managed clusters).
• If data is lost, restore Secrets from etcd backup or GitOps source.

Operations Checklist

Use this checklist during incident response or routine audits:

• [ ] Inventory: kubectl get secrets -n <ns> -o wide captured; Secret types, ages, and labels documented.
• [ ] Reference Mapping: Every Deployment/StatefulSet/Pod spec lists envFrom.secretRef or volumeMounts with exact Secret names and keys.
• [ ] Dry-Run Validation: All Secret changes generated with --dry-run=client -o yaml and committed to Git before apply.
• [ ] Rotation Drill: Quarterly test of password rotation using the v2 Secret + rollout pattern; measure MTTR.
• [ ] External Secrets Health: kubectl get externalsecret -A shows all Ready=True; last sync < 1 hour ago.
• [ ] Certificate Monitoring: kubectl get certificates -A (cert-manager) or custom script checks notAfter < 30 days; alert fires.
• [ ] RBAC Audit: kubectl auth can-i get secrets -n <ns> --as=system:serviceaccount:<ns>:<sa> returns yes only for required workloads.
• [ ] Backup Verification: SealedSecrets or Velero backup of Secrets namespace tested in staging cluster within last 30 days.
• [ ] Documentation: Runbook links to this article, includes cluster-specific commands (e.g., managed cluster node debug procedure).

Conclusion

Kubernetes Secrets troubleshooting is not a single command but a disciplined loop: observe the live state, compare it against the declared intent, isolate the smallest change that closes the gap, and verify the result before declaring success. The most costly incidents come from skipping the inventory step—assuming the Secret exists, has the right keys, and is mounted correctly—only to discover a silent key mismatch or a stale cache after a rotation. By treating Secrets as versioned artifacts, validating them at rest and at runtime, and rehearsing rotation and recovery, you turn a frequent source of outages into a predictable, auditable process. Start your next audit by running the inventory commands in this guide against one critical namespace; the gaps you find will be the ones that matter.