## Intro

Kubernetes Secrets store credentials, tokens, API keys, and other sensitive configuration. Treat them as crown jewels. This guide shows how to harden Secrets in real clusters with practical YAML and kubectl commands. You will learn to:

- Inventory and classify Secrets by sensitivity and owner

- Enable encryption at rest for Secrets in the API server data store

- Apply least‑privilege RBAC and dedicated service accounts per workload

- Consume Secrets via read‑only file mounts instead of environment variables

- Add namespace‑level default‑deny network policies with explicit allows

- Run quick, low‑risk checks to validate hardening

## Workflow Overview

Apply the steps namespace by namespace:

- Inventory and classify

- Encrypt at rest

- Lock down RBAC and service accounts

- Use Secrets safely in pods

- Contain network exposure

- Add checks and rotation

### 1) Inventory and classify Secrets

List all Secrets and tag owners:

kubectl get secrets -A -o wide 
 Add consistent labels for querying:

kubectl label secret db-credentials owner=payments purpose=database -n team-apps --overwrite 
 Create a new Secret with clear naming and scope:

kubectl create secret generic db-credentials \
 --from-literal=username=app \
 --from-literal=password='S3cr3t!' \
 -n team-apps 
 Tip: prefer one Secret per application or component rather than shared cross‑app Secrets. This limits blast radius and simplifies RBAC.

### 2) Encrypt Secrets at rest

Secrets reside in the API server data store. Enable encryption so they are encrypted on disk.

Example EncryptionConfiguration (aescbc provider):

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
 - secrets
 providers:
 - aescbc:
 keys:
 - name: key1
 secret: <base64-encoded-32-byte-key>
 - identity: {} 
 High‑level steps:

- Place the file on each control‑plane node.

- Start the API server with --encryption-provider-config=/path/to/config.yaml .

- Restart the API server and run kubectl get secrets --all-namespaces -o json | kubectl replace -f - to re‑encrypt existing data.

In managed clusters, enable the platform‑provided “Secrets encryption” setting.

### 3) Lock down RBAC and service accounts

Grant read access only to the workloads that need a specific Secret.

ServiceAccount that does not mount the default token:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: web-sa
 namespace: team-apps
automountServiceAccountToken: false 
 Role allowing get on a single Secret:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: read-db-credentials
 namespace: team-apps
rules:
- apiGroups: ["
- resources: ["secrets
- resourceNames: ["db-credentials
- verbs: ["get
auth can-i get secret db-credentials \
 --as=system:serviceaccount:team-apps:web-sa \
 -n team-apps 

### 4) Use Secrets safely in pods

 Mount Secrets as read‑only files; avoid environment variables which can leak into logs and /proc .

Pod spec (Deployment excerpt):

apiVersion: apps/v1
kind: Deployment
metadata:
 name: web
 namespace: team-apps
spec:
 replicas: 1
 selector:
 matchLabels:
 app: web
 template:
 metadata:
 labels:
 app: web
 spec:
 serviceAccountName: web-sa
 automountServiceAccountToken: false
 containers:
 - name: app
 image: your-registry/web:1.0.0
 volumeMounts:
 - name: db-secret
 mountPath: /var/run/secrets/app
 readOnly: true
 env:
 - name: DB_USER_FILE
 value: /var/run/secrets/app/username
 - name: DB_PASS_FILE
 value: /var/run/secrets/app/password
 volumes:
 - name: db-secret
 secret:
 secretName: db-credentials 
 Application reads the files at startup ( cat $DB_USER_FILE ). If environment variables are unavoidable, scope them to the minimal containers and never use envFrom with an entire Secret.

Make Secrets immutable to prevent accidental in‑place changes:

apiVersion: v1
kind: Secret
metadata:
 name: db-credentials
 namespace: team-apps
type: Opaque
immutable: true
data:
 username: YXBw
 password: UzNjcjN0IQ== 
 Rotate by creating db-credentials-v2 , updating the Deployment to reference it, rolling out, then deleting the old Secret after verification.

### 5) Contain network exposure

Network policies don’t protect Secrets directly, but they limit lateral movement.

Default deny all ingress and egress:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-all
 namespace: team-apps
spec:
 podSelector: {}
 policyTypes: [Ingress, Egress] 
 Allow DNS egress (adjust selectors for your cluster):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-dns-egress
 namespace: team-apps
spec:
 podSelector: {}
 policyTypes: [Egress]
 egress:
 - to:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: kube-system
 podSelector:
 matchLabels:
 k8s-app: kube-dns
 ports:
 - protocol: UDP
 port: 53
 - protocol: TCP
 port: 53 
 Add explicit allow policies for each required dependency.

### 6) Add quick checks and audits

Run these lightweight checks regularly.

Who can read which Secrets

kubectl auth can-i --list \
 --as=system:serviceaccount:team-apps:web-sa \
 -n team-apps | grep secrets || true 
 Pods that reference Secrets via env vars (requires jq ):

kubectl get pods -A -o json | jq -r '
 .items[]
 | select((.spec.containers // [])
 | map(.env // [] | map(has("valueFrom" ) and .valueFrom.secretKeyRef != null) | any) | any)
 | [.metadata.namespace, .metadata.name] | @tsv' 
 Secrets that are not immutable :

kubectl get secrets -A -o json | jq -r '
 .items[] | select(.immutable != true) | [.metadata.namespace, .metadata.name] | @tsv' 
 Wide‑scoped Roles/ClusterRoles that can read many Secrets :

kubectl get role,clusterrole -A -o yaml | grep -n "resources: \[secrets\]" || true 
 Confirm encryption at rest – inspect the API server flag --encryption-provider-config on control‑plane nodes or the managed‑service setting.

## Local Pilot Plan

Start small, measure, then expand.

Scope

- Namespace: team-apps

- Workload: a single Deployment web that needs db-credentials

Success criteria

- Only web-sa can get Secret db-credentials in team-apps

- No pods in team-apps consume Secrets via environment variables

- Namespace has a default‑deny network policy with explicit allows

Steps

- Create namespace and service account

kubectl create namespace team-apps
 kubectl apply -f - <<'EOF'
 apiVersion: v1
 kind: ServiceAccount
 metadata:
 name: web-sa
 namespace: team-apps
 automountServiceAccountToken: false
 EOF 

- Create the Secret

 kubectl create secret generic db-credentials \
 --from-literal=username=app \
 --from-literal=password='S3cr3t!' \
 -n team-apps 

- Apply least‑privilege RBAC

 kubectl apply -f - <<'EOF'
 apiVersion: rbac.authorization.k8s.io/v1
 kind: Role
 metadata:
 name: read-db-credentials
 namespace: team-apps
 rules:
 - apiGroups: ["
 resources: ["secrets"
 resourceNames: ["db-credentials"
 verbs: ["get
auth can-i get secret db-credentials \
 --as=system:serviceaccount:team-apps:web-sa \
 -n team-apps 

- Deploy the app mounting Secrets as files

 kubectl apply -f - <<'EOF'
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: web
 namespace: team-apps
 spec:
 replicas: 1
 selector:
 matchLabels:
 app: web
 template:
 metadata:
 labels:
 app: web
 spec:
 serviceAccountName: web-sa
 automountServiceAccountToken: false
 containers:
 - name: app
 image: your-registry/web:1.0.0
 volumeMounts:
 - name: db-secret
 mountPath: /var/run/secrets/app
 readOnly: true
 env:
 - name: DB_USER_FILE
 value: /var/run/secrets/app/username
 - name: DB_PASS_FILE
 value: /var/run/secrets/app/password
 volumes:
 - name: db-secret
 secret:
 secretName: db-credentials
 EOF 

- Apply default‑deny and allow DNS

 kubectl apply -f - <<'EOF'
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
 name: default-deny-all
 namespace: team-apps
 spec:
 podSelector: {}
 policyTypes: [Ingress, Egress]
 ---
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
 name: allow-dns-egress
 namespace: team-apps
 spec:
 podSelector: {}
 policyTypes: [Egress]
 egress:
 - to:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: kube-system
 podSelector:
 matchLabels:
 k8s-app: kube-dns
 ports:
 - protocol: UDP
 port: 53
 - protocol: TCP
 port: 53
 EOF 

- Validate

 # Only web-sa can read db-credentials
 kubectl auth can-i get secret db-credentials \
 --as=system:serviceaccount:team-apps:web-sa \
 -n team-apps

 # Default SA should not
 kubectl auth can-i get secret db-credentials \
 --as=system:serviceaccount:team-apps:default \
 -n team-apps

 # No pods consume Secrets via env vars
 kubectl get pods -n team-apps -o json | jq -r '
 .items[]
 | select((.spec.containers // [])
 | map(.env // [] | map(has("valueFrom" ) and .valueFrom.secretKeyRef != null) | any) | any)
 | .metadata.name' | wc -l

 # Network policies exist
 kubectl get networkpolicy -n team-apps 

- Document rotation for db-credentials

- Create db-credentials-v2

- Update Deployment to reference db-credentials-v2

- Roll pods and verify

- Delete db-credentials after validation

 Rollback plan

- Keep the previous Secret until verification completes.

- If rollout fails, revert the Deployment to the prior Secret reference and re‑run tests.

## Conclusion

You now have a practical path to harden Kubernetes Secrets: inventory, encrypt at rest, enforce least privilege, keep Secrets out of environment variables, restrict network exposure, and verify with repeatable checks. Pilot in one namespace, measure, then scale out.

Track a few core metrics as you expand:

- Secrets count per namespace and presence of owner labels

- Roles that can read arbitrary Secrets vs. specific‑name Roles

- Pods consuming Secrets via env vars vs. file mounts

- Namespaces covered by default‑deny network policies

A staged approach keeps changes safe and observable while improving your security posture step by step.