Intro
Kubernetes Secrets hold credentials, tokens, API keys, and other sensitive configuration. Treat them as crown jewels. This guide shows how to harden Secrets in real clusters with practical YAML and kubectl commands. You will learn how to reduce blast radius with RBAC, keep Secrets out of pod environments, turn on encryption at rest, restrict network exposure, and add simple checks you can run anytime.
What you will do:
- Inventory and classify Secrets by sensitivity and owner
- Enable encryption at rest for Secrets in the API server data store
- Apply least-privilege RBAC and service accounts per workload
- Consume Secrets via files rather than env vars wherever possible
- Add namespace-level default-deny network policies with explicit allows
- Run quick, low-risk checks to validate hardening
Workflow Overview
Use a staged flow you can apply namespace by namespace: 1) inventory and classify, 2) encrypt at rest, 3) lock down RBAC and service accounts, 4) use Secrets safely in pods, 5) contain network exposure, 6) add checks and rotation.
1) Inventory and classify Secrets
Start by listing Secrets across namespaces and tagging owners.
List Secrets with types:
kubectl get secrets -A -o wide
Optionally add consistent labels so you can query by owner and purpose:
kubectl label secret db-credentials owner=payments purpose=database -n team-apps --overwrite
Create new Secrets with clear naming and scoping:
kubectl create secret generic db-credentials \
--from-literal=username=app \
--from-literal=password='S3cr3t!' \
-n team-apps
Tip: prefer dedicated Secrets per application or component over shared cross-app Secrets. This constrains blast radius and access control.
2) Encrypt Secrets at rest
Secrets are stored in the API server data store. Enable encryption at rest so that stored Secrets are encrypted on disk.
Example EncryptionConfiguration (aescbc provider):
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <base64-encoded-32-byte-key>
- identity: {}
High-level steps:
- Place the EncryptionConfiguration file on control plane nodes.
- Configure the API server to use it (for example via --encryption-provider-config).
- Restart the API server and re-encrypt existing data if required.
In managed clusters, use the platform setting that enables encryption for Secrets.
3) Lock down RBAC and service accounts
Grant read access to Secrets only to the workloads that need them, and only to specific Secret names when possible.
ServiceAccount with token not mounted by default:
apiVersion: v1
kind: ServiceAccount
metadata:
name: web-sa
namespace: team-apps
automountServiceAccountToken: false
Role that permits reading only one Secret by name:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: read-db-credentials
namespace: team-apps
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["db-credentials"]
verbs: ["get"]
RoleBinding for that ServiceAccount:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: web-sa-read-db-credentials
namespace: team-apps
subjects:
- kind: ServiceAccount
name: web-sa
namespace: team-apps
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: read-db-credentials
Quick check:
kubectl auth can-i get secret db-credentials \
--as=system: serviceaccount: team-apps: web-sa \
-n team-apps
4) Use Secrets safely in pods
Prefer mounting Secrets as files over exposing them as environment variables. Env vars can leak into logs, crash reports, and /proc for debugging tools.
Pod spec mounting a Secret as read-only files:
apiVersion: v1
kind: Pod
metadata:
name: web
namespace: team-apps
spec:
serviceAccountName: web-sa
automountServiceAccountToken: false
containers:
- name: app
image: your-registry/web:1.0.0
volumeMounts:
- name: db-secret
mountPath: /var/run/secrets/app
readOnly: true
env:
- name: DB_USER_FILE
value: /var/run/secrets/app/username
- name: DB_PASS_FILE
value: /var/run/secrets/app/password
volumes:
- name: db-secret
secret:
secretName: db-credentials
In your application, read the file contents at startup rather than reading env vars. If you must use env vars, scope them to the minimal set of containers that truly need them and avoid broad projections like envFrom with whole Secrets.
Immutable Secrets prevent accidental in-place changes:
apiVersion: v1
kind: Secret
metadata:
name: db-credentials
namespace: team-apps
type: Opaque
immutable: true
data:
username: YXBw
password: UzNjcjN0IQ==
To rotate, create a new Secret (for example db-credentials-v2), update the Deployment to reference it, then delete the old one after verification.
5) Contain network exposure
Network policies do not protect Secrets directly, but they limit lateral movement and reduce the chance that a compromised pod can exfiltrate data.
Default deny all ingress and egress in a namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: team-apps
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
Allow egress to DNS only (adjust selectors to match your cluster):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: team-apps
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
Then add explicit allow policies for required app dependencies.
6) Add quick checks and audits
Run these lightweight checks regularly.
Who can read which Secrets:
# List effective permissions for a service account
kubectl auth can-i --list \
--as=system: serviceaccount: team-apps: web-sa \
-n team-apps | grep secrets || true
Find pods that reference Secrets via env vars (uses jq):
kubectl get pods -A -o json | jq -r '
.items[]
| select((.spec.containers // [])
| map(.env // [] | map(has("valueFrom") and .valueFrom.secretKeyRef != null) | any)
| any)
| [.metadata.namespace, .metadata.name] | @tsv'
List Secrets that are not immutable:
kubectl get secrets -A -o json | jq -r '
.items[] | select(.immutable != true) | [.metadata.namespace, .metadata.name] | @tsv'
Detect wide-scoped Roles that can read many Secrets:
kubectl get role, clusterrole -A -o yaml | grep -n "resources: \[secrets\]" -n || true
Confirm encryption at rest configuration by inspecting the API server flags and encryption provider config on control plane nodes or the equivalent managed setting.
Local Pilot Plan
Start small so you can measure progress and reduce risk.
Scope
- Namespace: team-apps
- Workload: a single Deployment named web that needs db-credentials
Success criteria
- Only service account web-sa can get Secret db-credentials in team-apps
- No pods in team-apps consume Secrets via env vars
- Namespace has default-deny network policy with explicit allows
Steps
- Create namespace and service account
kubectl create namespace team-apps
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: web-sa
namespace: team-apps
automountServiceAccountToken: false
EOF
- Create the Secret
kubectl create secret generic db-credentials \
--from-literal=username=app \
--from-literal=password='S3cr3t!' \
-n team-apps
- Apply least-privilege RBAC
kubectl apply -f - <<'EOF'
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: read-db-credentials
namespace: team-apps
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["db-credentials"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: web-sa-read-db-credentials
namespace: team-apps
subjects:
- kind: ServiceAccount
name: web-sa
namespace: team-apps
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: read-db-credentials
EOF
- Deploy the app mounting Secrets as files
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
namespace: team-apps
spec:
replicas: 1
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
serviceAccountName: web-sa
automountServiceAccountToken: false
containers:
- name: app
image: your-registry/web:1.0.0
volumeMounts:
- name: db-secret
mountPath: /var/run/secrets/app
readOnly: true
env:
- name: DB_USER_FILE
value: /var/run/secrets/app/username
- name: DB_PASS_FILE
value: /var/run/secrets/app/password
volumes:
- name: db-secret
secret:
secretName: db-credentials
EOF
- Apply default-deny and allow DNS
kubectl apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: team-apps
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-egress
namespace: team-apps
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
EOF
- Validate
# Only web-sa can read db-credentials
kubectl auth can-i get secret db-credentials \
--as=system: serviceaccount: team-apps: web-sa \
-n team-apps
# A random default SA should not
kubectl auth can-i get secret db-credentials \
--as=system: serviceaccount: team-apps: default \
-n team-apps
# No pods in the namespace consume Secrets via env vars
kubectl get pods -n team-apps -o json | jq -r '
.items[]
| select((.spec.containers // [])
| map(.env // [] | map(has("valueFrom") and .valueFrom.secretKeyRef != null) | any)
| any)
| .metadata.name' | wc -l
# Network policy objects exist
kubectl get networkpolicy -n team-apps
- Document rotation for db-credentials
- Create db-credentials-v2
- Update Deployment to reference db-credentials-v2
- Roll pods and verify
- Delete db-credentials after validation
Rollback plan
- Keep previous Secret until verification is complete
- If the rollout fails, revert the Deployment to the prior Secret reference and re-run tests
Conclusion
You now have a practical path to harden Kubernetes Secrets: inventory, encrypt at rest, enforce least privilege, keep Secrets out of env vars, restrict network exposure, and verify with repeatable checks. Pilot in one namespace, measure, and then scale out.
As you expand, track a few core metrics:
- Count of Secrets per namespace and owner labels present
- Number of Roles that can read arbitrary Secrets vs specific names
- Pods consuming Secrets via env vars vs file mounts
- Namespaces covered by default-deny network policies
A staged approach keeps changes safe and observable while improving your security posture step by step.