Kubernetes Secrets security hardening with practical examples

Intro

Kubernetes Secrets store credentials, tokens, API keys, and other sensitive configuration. Treat them as crown jewels. This guide shows how to harden Secrets in real clusters with practical YAML and kubectl commands. You will learn to:

Inventory and classify Secrets by sensitivity and owner
Enable encryption at rest for Secrets in the API server data store
Apply least‑privilege RBAC and dedicated service accounts per workload
Consume Secrets via read‑only file mounts instead of environment variables
Add namespace‑level default‑deny network policies with explicit allows
Run quick, low‑risk checks to validate hardening

Quick check 1 of 2

Which of the following is a recommended alternative to using Secrets for a cloud-native component that needs to authenticate to another application within the same Kubernetes cluster?

In reference passage [4], it states that if your cloud-native component needs to authenticate to another application that you know is running within the same Kubernetes cluster, you can use a ServiceAccount and its tokens to identify your client.

Workflow Overview

Apply the steps namespace by namespace:

Inventory and classify
Encrypt at rest
Lock down RBAC and service accounts
Use Secrets safely in pods
Contain network exposure
Add checks and rotation

1) Inventory and classify Secrets

List all Secrets and tag owners:

kubectl get secrets -A -o wide

Add consistent labels for querying:

kubectl label secret db-credentials owner=payments purpose=database -n team-apps --overwrite

Create a new Secret with clear naming and scope:

kubectl create secret generic db-credentials \
  --from-literal=username=app \
  --from-literal=password='S3cr3t!' \
  -n team-apps

Tip: prefer one Secret per application or component rather than shared cross‑app Secrets. This limits blast radius and simplifies RBAC.

2) Encrypt Secrets at rest

Secrets reside in the API server data store. Enable encryption so they are encrypted on disk.

Example EncryptionConfiguration (aescbc provider):

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-32-byte-key>
  - identity: {}

High‑level steps:

Place the file on each control‑plane node.
Start the API server with --encryption-provider-config=/path/to/config.yaml.
Restart the API server and run kubectl get secrets --all-namespaces -o json | kubectl replace -f - to re‑encrypt existing data.

In managed clusters, enable the platform‑provided “Secrets encryption” setting.

3) Lock down RBAC and service accounts

Grant read access only to the workloads that need a specific Secret.

ServiceAccount that does not mount the default token:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: web-sa
  namespace: team-apps
automountServiceAccountToken: false

Role allowing get on a single Secret:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: read-db-credentials
  namespace: team-apps
rules:
- apiGroups: ["
- resources: ["secrets
- resourceNames: ["db-credentials
- verbs: ["get
auth can-i get secret db-credentials \
  --as=system:serviceaccount:team-apps:web-sa \
  -n team-apps

4) Use Secrets safely in pods

Mount Secrets as read‑only files; avoid environment variables which can leak into logs and /proc.

Pod spec (Deployment excerpt):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: team-apps
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      serviceAccountName: web-sa
      automountServiceAccountToken: false
      containers:
      - name: app
        image: your-registry/web:1.0.0
        volumeMounts:
        - name: db-secret
          mountPath: /var/run/secrets/app
          readOnly: true
        env:
        - name: DB_USER_FILE
          value: /var/run/secrets/app/username
        - name: DB_PASS_FILE
          value: /var/run/secrets/app/password
      volumes:
      - name: db-secret
        secret:
          secretName: db-credentials

Application reads the files at startup (cat $DB_USER_FILE). If environment variables are unavoidable, scope them to the minimal containers and never use envFrom with an entire Secret.

Make Secrets immutable to prevent accidental in‑place changes:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: team-apps
type: Opaque
immutable: true
data:
  username: YXBw
  password: UzNjcjN0IQ==

Rotate by creating db-credentials-v2, updating the Deployment to reference it, rolling out, then deleting the old Secret after verification.

5) Contain network exposure

Network policies don’t protect Secrets directly, but they limit lateral movement.

Default deny all ingress and egress:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: team-apps
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

Allow DNS egress (adjust selectors for your cluster):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: team-apps
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Add explicit allow policies for each required dependency.

6) Add quick checks and audits

Run these lightweight checks regularly.

Who can read which Secrets

kubectl auth can-i --list \
  --as=system:serviceaccount:team-apps:web-sa \
  -n team-apps | grep secrets || true

Pods that reference Secrets via env vars (requires jq):

kubectl get pods -A -o json | jq -r '
  .items[]
  | select((.spec.containers // [])
    | map(.env // [] | map(has("valueFrom" ) and .valueFrom.secretKeyRef != null) | any) | any)
  | [.metadata.namespace, .metadata.name] | @tsv'

Secrets that are not immutable:

kubectl get secrets -A -o json | jq -r '
  .items[] | select(.immutable != true) | [.metadata.namespace, .metadata.name] | @tsv'

Wide‑scoped Roles/ClusterRoles that can read many Secrets:

kubectl get role,clusterrole -A -o yaml | grep -n "resources: \[secrets\]" || true

Confirm encryption at rest – inspect the API server flag --encryption-provider-config on control‑plane nodes or the managed‑service setting.

Quick check 2 of 2

What is a key reason to avoid using environment variables to inject secrets into pods?

Reference passage [1] states: 'Please note that the environment variable method might be more prone to leakage due to crash dumps in logs and the non-confidential nature of environment variable in Linux, as opposed to the permission mechanism on files.'

Local Pilot Plan

Start small, measure, then expand.

Scope

Namespace: team-apps
Workload: a single Deployment web that needs db-credentials

Success criteria

Only web-sa can get Secret db-credentials in team-apps
No pods in team-apps consume Secrets via environment variables
Namespace has a default‑deny network policy with explicit allows

Steps

Create namespace and service account

   kubectl create namespace team-apps
   kubectl apply -f - <<'EOF'
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: web-sa
     namespace: team-apps
   automountServiceAccountToken: false
   EOF

Create the Secret

   kubectl create secret generic db-credentials \
     --from-literal=username=app \
     --from-literal=password='S3cr3t!' \
     -n team-apps

Apply least‑privilege RBAC

   kubectl apply -f - <<'EOF'
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: read-db-credentials
     namespace: team-apps
   rules:
   - apiGroups: ["
     resources: ["secrets"
     resourceNames: ["db-credentials"
     verbs: ["get
auth can-i get secret db-credentials \
   --as=system:serviceaccount:team-apps:web-sa \
   -n team-apps

Deploy the app mounting Secrets as files

   kubectl apply -f - <<'EOF'
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: web
     namespace: team-apps
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: web
     template:
       metadata:
         labels:
           app: web
       spec:
         serviceAccountName: web-sa
         automountServiceAccountToken: false
         containers:
         - name: app
           image: your-registry/web:1.0.0
           volumeMounts:
           - name: db-secret
             mountPath: /var/run/secrets/app
             readOnly: true
           env:
           - name: DB_USER_FILE
             value: /var/run/secrets/app/username
           - name: DB_PASS_FILE
             value: /var/run/secrets/app/password
         volumes:
         - name: db-secret
           secret:
             secretName: db-credentials
   EOF

Apply default‑deny and allow DNS

   kubectl apply -f - <<'EOF'
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: default-deny-all
     namespace: team-apps
   spec:
     podSelector: {}
     policyTypes: [Ingress, Egress]
   ---
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: allow-dns-egress
     namespace: team-apps
   spec:
     podSelector: {}
     policyTypes: [Egress]
     egress:
     - to:
       - namespaceSelector:
           matchLabels:
             kubernetes.io/metadata.name: kube-system
         podSelector:
           matchLabels:
             k8s-app: kube-dns
       ports:
       - protocol: UDP
         port: 53
       - protocol: TCP
         port: 53
   EOF

Validate

   # Only web-sa can read db-credentials
   kubectl auth can-i get secret db-credentials \
     --as=system:serviceaccount:team-apps:web-sa \
     -n team-apps

   # Default SA should not
   kubectl auth can-i get secret db-credentials \
     --as=system:serviceaccount:team-apps:default \
     -n team-apps

   # No pods consume Secrets via env vars
   kubectl get pods -n team-apps -o json | jq -r '
     .items[]
     | select((.spec.containers // [])
       | map(.env // [] | map(has("valueFrom" ) and .valueFrom.secretKeyRef != null) | any) | any)
     | .metadata.name' | wc -l

   # Network policies exist
   kubectl get networkpolicy -n team-apps

Document rotation for db-credentials

Create db-credentials-v2
Update Deployment to reference db-credentials-v2
Roll pods and verify
Delete db-credentials after validation

Rollback plan

Keep the previous Secret until verification completes.
If rollout fails, revert the Deployment to the prior Secret reference and re‑run tests.

Conclusion

You now have a practical path to harden Kubernetes Secrets: inventory, encrypt at rest, enforce least privilege, keep Secrets out of environment variables, restrict network exposure, and verify with repeatable checks. Pilot in one namespace, measure, then scale out.

Track a few core metrics as you expand:

Secrets count per namespace and presence of owner labels
Roles that can read arbitrary Secrets vs. specific‑name Roles
Pods consuming Secrets via env vars vs. file mounts
Namespaces covered by default‑deny network policies

A staged approach keeps changes safe and observable while improving your security posture step by step.