## Intro Kubernetes Secrets store sensitive information such as passwords, tokens, and TLS keys. If a Secret is misconfigured, leaked, or unavailable, applications fail or credentials can be compromised. Monitoring Secrets is not about exposing their contents. It is about detecting when they are created, updated, deleted, or fail to mount correctly, and when access patterns look suspicious. This guide covers practical Secret monitoring for developers, DevOps engineers, SREs, and platform teams. It connects alerts, metrics, dashboards, and incident response to real commands, expected outputs, and recovery decisions. The focus is operational safety: observe before changing, limit blast radius, avoid logging secret values, verify outcomes, and document recovery paths. You will find concrete examples for Kubernetes audit logs, the Metrics Server, Prometheus, Alertmanager, Grafana, and a step-by-step incident response playbook. Every example assumes a working cluster with `kubectl` configured and, where noted, Prometheus or Grafana installed. ## Version and Environment Inventory Before monitoring Secrets, know your environment. Start with read-only observations to establish the current state. This prevents mistakes like applying a fix to the wrong cluster or assuming a feature is available when it is not. ### Check Kubernetes Version ```bash kubectl version --short ``` Expected output example: ```text Client Version: v1.28.2 Server Version: v1.28.4 ``` Some Secret monitoring features, such as audit log policies or `Immutable` Secrets, depend on the API server version. For example, `Immutable` Secrets became stable in Kubernetes 1.21. If your cluster is older, do not rely on that field. ### List Existing Secrets ```bash kubectl get secrets --all-namespaces ``` Example output: ```text NAMESPACE NAME TYPE DATA AGE default my-app-secret Opaque 2 3d kube-system default-token-abcde kubernetes.io/service-account-token 3 180d ``` Note the namespace, name, type, number of data keys, and age. A recently created Secret in a sensitive namespace may warrant a closer look. ### Inspect Without Revealing Values Never run `kubectl get secret -o yaml` casually, as it prints base64-encoded values that can be decoded. Instead, list keys only: ```bash kubectl get secret my-app-secret -n default -o jsonpath='{.data}' | jq 'keys' ``` Example output: ```text [ "password", "username" ] ``` This confirms the Secret exists and has the expected keys without exposing the data. ### Check Mount Status If an application uses a Secret, verify that it is mounted correctly. Find the pod: ```bash kubectl get pods -n default -l app=my-app ``` Then inspect volume mounts: ```bash kubectl describe pod -n default | grep -A 5 Mounts ``` Example output: ```text Mounts: /etc/app-secrets from secret-volume (ro) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xxxxx (ro) ``` If the mount point is missing, the Secret may not be referenced in the pod spec. ## Safe Configuration Path Configuration changes to Secret monitoring should be incremental. Start with read-only metrics and audit logs before enabling any blocking alerts. ### Enable Audit Logging Kubernetes audit logs record all API requests, including those to Secrets. Configure the API server with an audit policy file that logs Secret access at the `Metadata` level or higher. A minimal policy file: ```yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata resources: - group: "" resources: ["secrets"] ``` Apply the policy to the API server (this requires kube-apiserver flags `--audit-policy-file` and `--audit-log-path`). Then test by listing Secrets: ```bash kubectl get secrets ``` Check the audit log for entries: ```text {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"...","stage":"ResponseComplete","requestURI":"/api/v1/secrets","verb":"list","user":{"username":"admin"},"sourceIPs":["10.0.0.1"],"responseStatus":{"code":200}} ``` This gives you a trail without logging secret data. ### Use a Dedicated Service Account Create a dedicated Service Account for monitoring tools with least privilege. For example: ```yaml apiVersion: v1 kind: ServiceAccount metadata: name: secret-monitor namespace: monitoring --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: secret-monitor-role rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: secret-monitor-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: secret-monitor-role subjects: - kind: ServiceAccount name: secret-monitor namespace: monitoring ``` Apply it: ```bash kubectl apply -f secret-monitor-rbac.yaml ``` Test access from the Service Account: ```bash kubectl auth can-i list secrets --as=system:serviceaccount:monitoring:secret-monitor ``` Expected: `yes` ### Test in a Non-Production Namespace Before changing production, create a test Secret in a sandbox namespace and verify that your monitoring stack detects it. ## Verification and Diagnostics With basic observation in place, collect metrics and set up diagnostics that reveal the health of Secret usage. ### Metrics from the API Server The Kubernetes API server exposes metrics about requests, including Secret operations. If you have Prometheus scraping the API server, useful metrics include: - `apiserver_request_total{resource="secrets"}` - `apiserver_request_duration_seconds_bucket{resource="secrets"}` For example, a PromQL query to see the rate of Secret list requests over 5 minutes: ```promql rate(apiserver_request_total{resource="secrets", verb="list"}[5m]) ``` This can indicate unusual activity if the rate spikes. ### Check Pods with Mounted Secrets Use `kubectl` to find pods that mount a particular Secret. This helps diagnose why a Secret update is not reflected (pods often need a restart to pick up changes to mounted Secrets): ```bash kubectl get pods --all-namespaces -o json | jq -r '.items[] | select(.spec.volumes[]?.secret.secretName == "my-app-secret") | .metadata.namespace + "/" + .metadata.name' ``` Example output: ```text default/my-app-7d4f8b9c6-abcde default/my-app-7d4f8b9c6-fghij ``` ### Inspect Events Events often capture Secret-related issues, such as failed mounts: ```bash kubectl get events --all-namespaces --field-selector reason=FailedMount ``` Example output: ```text NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE default 2m Warning FailedMount pod/my-app-... MountVolume.SetUp failed for volume "secret-volume" : secret "my-app-secret" not found ``` This immediately tells you the Secret is missing. ### Use Prometheus Alert Rules Define alert rules for Secret operations. For example, alert if a Secret is deleted unexpectedly: ```yaml groups: - name: kubernetes-secrets rules: - alert: SecretDeleted expr: increase(apiserver_request_total{resource="secrets", verb="delete"}[10m]) > 0 for: 1m labels: severity: critical annotations: summary: "Secret deleted in the last 10 minutes" description: "A Secret delete request was observed. Investigate immediately." ``` Test the alert by deleting a test Secret and checking Alertmanager. ## Failure Modes and Recovery Secrets fail in predictable ways. Knowing the failure modes helps you detect and recover quickly. ### Failure Mode 1: Secret Not Found **Symptom**: Pods fail to start with `FailedMount` events referencing a missing Secret. **Diagnosis**: Check the namespace and Secret name: ```bash kubectl get secrets -n | grep ``` If not found, the Secret may have been deleted or created in a different namespace. **Recovery**: Recreate the Secret from a trusted source. If using version control or a secret manager, apply the manifest: ```bash kubectl apply -f secret.yaml ``` Then restart the affected pods: ```bash kubectl rollout restart deployment ``` ### Failure Mode 2: Wrong Secret Data **Symptom**: Application connects to a database but gets authentication errors. The Secret exists but contains incorrect values. **Diagnosis**: Compare the data keys and, without printing values, check the base64 encoding or hash. If possible, check the application logs for clues without exposing secrets. **Recovery**: Update the Secret with correct values using a secure method. For example, create from literals: ```bash kubectl create secret generic my-app-secret --from-literal=username=correct-user --from-literal=password='correct-password' --dry-run=client -o yaml | kubectl apply -f - ``` Then restart the pods. ### Failure Mode 3: Secret Rotation Not Reflected **Symptom**: After rotating a secret in an external system, the application still uses old credentials. This often happens because pods cache Secrets. **Diagnosis**: Check the pod's start time and the Secret's age: ```bash kubectl get pod -o jsonpath='{.metadata.creationTimestamp}' kubectl get secret -o jsonpath='{.metadata.creationTimestamp}' ``` If the pod is older than the Secret update, it is using the old mounted volume. **Recovery**: Restart the pods to remount the updated Secret: ```bash kubectl rollout restart deployment ``` If using a Secret with `immutable: true`, you must delete and recreate the pod or use a new Secret name. ### Failure Mode 4: Unauthorized Access or Leak **Symptom**: Audit logs show unexpected `get` or `list` requests for Secrets from an unknown user or IP. **Diagnosis**: Review audit logs: ```text {"verb":"get","resource":"secrets","user":{"username":"system:serviceaccount:default:compromised-sa"},"sourceIPs":["203.0.113.50"]} ``` **Recovery**: Immediately revoke the Service Account or user credentials, rotate affected Secrets, and investigate the source IP. Apply a network policy to restrict access to the API server if needed. ## Dashboard and Continuous Monitoring A dashboard gives an at-a-glance view of Secret health and activity. Use Grafana with Prometheus data source. ### Example Grafana Panels **Panel 1: Secrets Created/Deleted (last 24 hours)** ```promql sum(increase(apiserver_request_total{resource="secrets", verb="create"}[24h])) by (namespace) ``` **Panel 2: Secret Read Rate by User** ```promql sum by (user) (rate(apiserver_request_total{resource="secrets", verb="get"}[5m])) ``` **Panel 3: Pods with Failed Secret Mounts** This requires a custom exporter or using kube-state-metrics. Kube-state-metrics exposes `kube_pod_container_status_waiting_reason` for `CrashLoopBackOff`, but for Secret mount failures, you may need to parse Events or use a custom metric. Alternatively, use the following Elasticsearch or Loki log-based dashboard if you collect events. ### Example Alert: Failed Secret Mount If you collect Kubernetes events into Prometheus via an exporter like `eventrouter`, you can create an alert: ```yaml - alert: SecretMountFailed expr: increase(kube_event_count{reason="FailedMount", involved_object_kind="Pod"}[10m]) > 0 for: 5m labels: severity: warning annotations: summary: "Pods are failing to mount a Secret" ``` ### Using kube-state-metrics Kube-state-metrics provides a metric `kube_secret_created` and `kube_secret_info`. You can alert on unexpected new Secrets: ```promql increase(kube_secret_created[1h]) > 3 ``` This could be a sign of misconfiguration or an attack. ## Operations Checklist Use this checklist as part of your weekly operations review. Assign each item to an owner and set a revisit frequency. For example, "Priya Shah, Engineering Lead" owns the Secrets monitoring review, and the review happens every Monday morning. ### Daily Checks (Automated) - [ ] Alertmanager has no active critical alerts for Secret operations. - [ ] Grafana dashboard shows no unexpected spikes in Secret reads. - [ ] Audit log sample shows no unauthorized Secret access attempts. - [ ] All pods with mounted Secrets are in Running state and not crash looping. ### Weekly Checks (Manual Review) - [ ] Review the list of Secrets created or deleted in the past week. Owner: Priya Shah, Engineering Lead. Revisit weekly. - [ ] Confirm that rotated Secrets have been updated and pods restarted. Owner: Marcus Chen, DevOps Engineer. - [ ] Check for stale Secrets not used by any pod. Use: ```bash kubectl get secrets --all-namespaces -o json | jq -r '.items[] | select(.metadata.ownerReferences == null) | .metadata.namespace + "/" + .metadata.name' ``` Then verify if they are needed. - [ ] Review RBAC permissions for Secret access and remove unused Service Accounts. Owner: Aisha Patel, Security Engineer. ### Monthly Audit (Security Team) - [ ] Audit all audit log rules for completeness and adjust as necessary. - [ ] Perform a simulated Secret leak exercise to test incident response. - [ ] Rotate all static Secrets that are older than 90 days. - [ ] Assess external secret management integration (e.g., Vault, AWS Secrets Manager) if not already in use. ## Common Pitfalls and How to Avoid Them ### Pitfall 1: Logging Secret Values **Why it happens**: Developers use `kubectl get secret -o yaml` for debugging and the output is captured in logs or CI output. **How to avoid**: Never use `-o yaml` on Secrets. Use `kubectl get secret -o jsonpath='{.data}'` to list keys only. Redact secrets in logs and set CI systems to mask sensitive values. ### Pitfall 2: Assuming Secret Updates Propagate Instantly **Why it happens**: Teams rotate a Secret and expect all pods to use the new value immediately. However, mounted Secrets are only updated periodically by the kubelet (if using environment variables, they never update). **How to avoid**: Treat Secret updates as requiring a pod restart. Use a deployment rollout or a tool like Reloader to automate restarts. ### Pitfall 3: Not Monitoring for Unauthorized Access **Why it happens**: Monitoring focuses on pod health but not on API server audit logs, so someone may exfiltrate Secrets unnoticed. **How to avoid**: Enable audit logging with `Metadata` level for Secrets, and set up alerts for unusual access patterns (e.g., a user accessing many Secrets in a short time). ### Pitfall 4: Overly Broad RBAC **Why it happens**: To simplify, teams grant `get` and `list` on all Secrets to many Service Accounts. **How to avoid**: Use least privilege. Only grant Secret read access to components that absolutely need it. Use separate Service Accounts per application and avoid cluster-wide Secret read permissions. ### Pitfall 5: Alert Fatigue **Why it happens**: Alerts are too noisy, such as alerting on every Secret creation, causing teams to ignore them. **How to avoid**: Tune alert thresholds based on normal behavior. Use `for` clauses to require the condition to persist, and distinguish between warning and critical severity. ## Conclusion Kubernetes Secret monitoring is essential for security and reliability. It requires a combination of audit logging, metrics, alerting, and dashboards, all wrapped in a safe operational process. Start with read-only observations, implement alerts gradually, and always protect secret values from exposure. By following the practical steps in this guide, you can detect failures, unauthorized access, and configuration drift before they cause incidents. Assign clear ownership for monitoring tasks and review them regularly to keep the system effective. Use the operations checklist to ensure nothing is missed, and continuously refine your alerts as your cluster grows. ## Final Words Monitoring Kubernetes Secrets does not have to be complex. Begin with the basics: know your environment, enable audit logs, set up Prometheus and Grafana, and create a handful of meaningful alerts. As your confidence grows, expand into automated rotation and external secret management. The key is to make monitoring a habit, not an afterthought.