Kubernetes Encrypt Secret Data at Rest: A Production Operations Checklist with Practical Examples

Intro

Kubernetes Secrets are the standard way to store sensitive data such as passwords, API tokens, and TLS keys. By default, Secrets are stored unencrypted in etcd, the cluster's backing store. Anyone with access to etcd (or its backups) can read them, which is a serious security risk. Encrypting Secret data at rest is a critical control for production clusters.

This article provides a practical operations checklist for enabling and verifying encryption of Kubernetes Secrets at rest. It is aimed at developers, DevOps consultants, and technical startup teams who need to move from an observed problem to a verified result. The focus is on operational safety: observe before changing, limit the blast radius, use placeholders instead of real secrets, verify the result, and document recovery steps.

We'll cover version and environment inventory, safe configuration of the kube-apiserver encryption configuration, verification and diagnostics, failure modes and recovery, and a concise operations checklist. Each section includes commands, expected output, failure signals, and recovery decisions.

Version and Environment Inventory

Before making any changes, you must understand your cluster's version and the configuration of the API server. Encryption at rest is configured directly on the kube-apiserver via the --encryption-provider-config flag. It is not a Kubernetes resource; it is a static file that the API server reads at startup.

Identify Kubernetes version and API server configuration

Run the following command to check the cluster version:

kubectl version --short

Expected output (example):

Client Version: v1.27.3
Server Version: v1.27.3

Encryption at rest is supported in all recent Kubernetes versions, but the configuration format may have evolved. The EncryptionConfiguration API is versioned: apiserver.config.k8s.io/v1 has been stable since v1.13, and v1 is recommended for current clusters. Check if your API server already has encryption configured by looking at the pod spec (if running as static pods) or the process arguments.

For a kubeadm cluster, the API server manifest is usually at /etc/kubernetes/manifests/kube-apiserver.yaml. For a managed cluster (EKS, GKE, AKS), you may not have direct access; you must use the provider's mechanisms (e.g., EKS encryption provider, GKE application-layer secrets encryption).

To see the API server command line in a running cluster, use:

kubectl -n kube-system get pod kube-apiserver-<node-name> -o jsonpath='{.spec.containers[0].command}'

Look for --encryption-provider-config. If absent, encryption is not enabled.

Prerequisites

You need cluster-admin permissions to modify the API server configuration and restart it.
Ensure you have access to the control plane node(s) or the ability to update the API server settings in your managed environment.
Back up the existing API server manifest or configuration before changes.
Have a local test environment (e.g., kind or minikube) to practice the procedure safely.

Read-only observation

Check the current encryption status by creating a test Secret and inspecting etcd directly (if possible) or using a debug container. For example, create a Secret:

kubectl create secret generic my-secret --from-literal=key=supersecret

Then, if you have access to etcd, you can query it (in a kubeadm cluster, etcd runs as a static pod):

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret

Expected output if unencrypted: you will see the literal string supersecret in the output, proving data is stored in plaintext.

If you cannot access etcd, you can inspect the API server logs for errors about encryption. Use kubectl logs -n kube-system kube-apiserver-<node-name> | grep -i encrypt to see if encryption is active.

Smallest justified change

The smallest change to enable encryption is to create an EncryptionConfiguration file and add the --encryption-provider-config flag to the API server. However, enabling encryption does not retroactively encrypt existing Secrets. You must rewrite all Secrets to trigger encryption, which we will cover later.

Verify the outcome

After enabling encryption and restarting the API server, create a new Secret and repeat the etcd inspection. You should see the data prefixed with k8s:enc:aescbc:v1: (if using AES-CBC) or another provider prefix, and no plaintext.

Practical Kubernetes check for Version and Environment Inventory:

Start with kubectl get pods -n kube-system -o wide to see control plane pods.
Use kubectl describe pod -n kube-system kube-apiserver-<node-name> to examine events and mounts.
Check kubectl logs -n kube-system kube-apiserver-<node-name> --previous for crash loop details.
Run kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp to see recent issues.

Always keep a local test small. Apply one manifest, inspect generated resources, and verify with kubectl port-forward or a local service type before moving to a cloud load balancer or ingress controller.

Safe Configuration Path

Now that you have inventoried the environment, you can safely configure encryption at rest. The key steps are:

Choose an encryption provider (e.g., AES-CBC, AES-GCM, KMS).
Create the EncryptionConfiguration file.
Update the API server manifest to reference it.
Restart the API server and verify it starts correctly.
Rewrite all existing Secrets to ensure they are encrypted.

Choosing an encryption provider

Kubernetes supports several providers:

aescbc: Recommended for encryption at rest. Uses AES-CBC with PKCS#7 padding. Requires a 32-byte key.
aesgcm: Uses AES-GCM. Note: GCM is non-deterministic, and etcd does not support watch on non-deterministic encryption in older versions. Prefer aescbc for etcd compatibility.
secretbox: Uses XSalsa20 and Poly1305. Also acceptable but less common.
kms: Uses an external Key Management Service (e.g., AWS KMS, GCP KMS, Azure Key Vault). Recommended for production because keys are managed externally and can be rotated.

For this checklist, we'll use aescbc as an example because it is simple and widely supported. In production, consider using a KMS provider for better key management.

Create the EncryptionConfiguration file

Create a file named encryption-config.yaml with the following content:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

You must generate a random 32-byte key and base64-encode it. Example:

head -c 32 /dev/urandom | base64

Expected output: a 44-character base64 string ending with =, e.g., dGhpcyBpcyBhIDMyIGJ5dGUga2V5IGZvciBkZW1vbmQ= (this is an example; generate your own).

Replace <base64-encoded-32-byte-key> with your generated key. The identity: {} provider as the second entry allows reading of existing unencrypted Secrets during migration. Once all Secrets are encrypted, you can remove the identity provider for stricter security.

Update the API server manifest

If you are using kubeadm, edit the manifest file on the control plane node:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add the following to the spec.containers[0].command array:

- --encryption-provider-config=/etc/kubernetes/encryption-config.yaml

Also add a volume mount and volume:

Under volumeMounts:

- name: encryption-config
  mountPath: /etc/kubernetes/encryption-config.yaml
  readOnly: true

Under volumes:

- name: encryption-config
  hostPath:
    path: /etc/kubernetes/encryption-config.yaml
    type: File

Place the encryption-config.yaml file on the control plane node at /etc/kubernetes/encryption-config.yaml with permissions 600 and owner root.

For high availability clusters with multiple control plane nodes, you must copy the file to all nodes and update each manifest.

Restart the API server

The kubelet will automatically restart the API server when the manifest changes. Monitor the restart:

kubectl get pods -n kube-system -w

Wait until the new API server pod is Running and Ready. Check logs for encryption-related messages:

kubectl logs -n kube-system kube-apiserver-<node-name> | grep -i encryption

Expected output: no errors. If you see errors like "invalid encryption config" or "failed to load encryption config", check file permissions and content.

Verify new Secrets are encrypted

Create a new Secret:

kubectl create secret generic test-encryption --from-literal=key=value123

Inspect etcd again (as shown earlier). You should see the encrypted form with prefix k8s:enc:aescbc:v1:key1: and no plaintext.

Rewrite all existing Secrets

Existing Secrets are not automatically re-encrypted. You must update them to trigger rewriting. Use this command:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

This command reads all Secrets and replaces them with the same data, causing the API server to re-encrypt them using the new provider. Be careful: this will overwrite any fields that are not in the current object, but for Secrets it is safe because the data is preserved.

After rewriting, verify that a previously existing Secret is now encrypted in etcd.

Practical Kubernetes check for Safe Configuration Path:

Use kubectl get events -n kube-system --watch during API server restart to catch failures.
If the API server fails to start, revert the manifest by removing the added flag and volume, then restart.
Keep a backup of the original manifest and the encryption config file.
Use kubectl rollout status for any deployments that depend on Secrets to ensure they still function.

Verification and Diagnostics

After enabling encryption, you need to verify that it works correctly and diagnose any issues. This section provides a systematic approach.

Verify the API server configuration

Check that the API server is using the encryption config:

kubectl -n kube-system get pod kube-apiserver-<node-name> -o yaml | grep -A5 -B5 encryption-provider-config

Expected output: the flag appears in the command.

Validate the encryption config file

You can use kube-apiserver with the --encryption-provider-config flag and a dry-run to validate the file without affecting the running cluster:

kube-apiserver --encryption-provider-config=/path/to/encryption-config.yaml --dry-run

This may require additional flags to avoid connection errors, but it can help detect malformed YAML or invalid provider names.

Test encryption of new Secrets

As described, create a test Secret and inspect etcd. Also, you can use the API server's audit logs to confirm encryption operations. If audit logging is enabled, look for encrypt or decrypt actions.

Ensure existing Secrets are encrypted

After rewriting all Secrets, you should verify that none remain in plaintext. You can query etcd for all secrets and check for the absence of plaintext values. For example, get all keys under /registry/secrets/ and search for known plaintext strings.

If you stored the plaintext before encryption, compare now.

Diagnostics: API server not starting

If the API server pod is in CrashLoopBackOff, check logs:

kubectl logs -n kube-system kube-apiserver-<node-name> --previous

Look for lines like:

F0710 12:00:00.000000 1 server.go:123] error loading encryption provider config: ...

This indicates a problem with the config file. Common issues:

File not found at the mount path.
Incorrect YAML syntax or API version.
Invalid base64 key length (must decode to 32 bytes for aescbc).
Permission denied on the file.

Fix the issue, save the file, and the API server should restart automatically.

Diagnostics: existing Secrets not encrypted after rewrite

If after running the rewrite command, etcd still shows plaintext for some Secrets, check the encryption config. The identity provider might be first in the list, or the resource list might miss secrets. Verify order: providers are tried in order, and the first provider that can encrypt is used. For reading, the API server tries providers until one succeeds. If identity is first, new writes will be unencrypted.

Practical Kubernetes check for Verification and Diagnostics:

Run kubectl get secrets --all-namespaces | wc -l before and after rewrite to ensure count is consistent.
Use kubectl get secret <name> -o yaml to compare data (base64-encoded) before and after; they should be identical.
Monitor API server metrics if enabled: apiserver_storage_transformation_operations_total and apiserver_storage_transformation_duration_seconds.
Check etcd metrics for encryption-related errors.

Failure Modes and Recovery

This section covers common failure scenarios and how to recover from them.

Failure: API server fails to start due to invalid encryption config

Symptom: API server pod is in CrashLoopBackOff, cluster inaccessible via kubectl.

Recovery:

SSH to the control plane node.
Edit the API server manifest (/etc/kubernetes/manifests/kube-apiserver.yaml) and remove the --encryption-provider-config flag and the volume mount/volume.
Save the file; kubelet will restart the API server without encryption.
Once cluster is accessible, fix the encryption config file.
Re-add the flag and volume, then save again.

Always have a backup of the manifest before making changes.

Failure: Unable to decrypt Secrets after key rotation

If you rotate encryption keys by adding a new key at the top of the keys list, the old key must remain in the list so that existing Secrets can be decrypted. If you remove the old key, the API server cannot decrypt previously encrypted Secrets, leading to failures when reading them.

Recovery: Restore the previous key to the keys list, restart API server, and then perform key rotation correctly:

Add new key as the first entry, keep old key as second.
Restart API server.
Rewrite all Secrets (kubectl get secrets --all-namespaces -o json | kubectl replace -f -).
Verify all Secrets are encrypted with the new key (you can check the prefix in etcd, which includes the key name).
Remove the old key from the config and restart API server.

Failure: Loss of encryption config file

If the encryption config file is lost (e.g., control plane node disk failure) and you have no backup, all Secrets encrypted with that key are unrecoverable. This is why KMS providers are recommended: they store keys externally.

Prevention: Always back up the encryption config file and the keys to a secure location. Store copies in a secrets manager or offline.

Failure: API server performance degradation

Encryption adds CPU overhead to API server requests that involve Secrets. For high-traffic clusters, this could increase latency. Monitor API server resource usage and consider using KMS with hardware-backed encryption or a more efficient provider. If performance is unacceptable, you may need to scale the API server or reduce Secret read/write frequency.

Failure: Incomplete rewrite of Secrets

If the rewrite command is interrupted (e.g., network issue), some Secrets may remain unencrypted. Run the rewrite command again; it is idempotent. To ensure complete encryption, remove the identity provider from the config after the rewrite. Without identity, the API server will only allow reads of encrypted Secrets, so if any plaintext remains, those Secrets will fail to read, revealing the problem.

Recovery: If you removed identity and some Secrets are unreadable, re-add identity temporarily, rewrite all Secrets again, then remove it.

Practical Kubernetes check for Failure Modes and Recovery:

Before any change, take an etcd snapshot if possible: ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-$(date +%Y%m%d).db.
Test the rollback procedure in a staging cluster.
Document the exact steps for your environment, including managed service specifics.
Use kubectl get secrets --all-namespaces -o json > secrets-backup.json to have a backup of all Secrets (but note that this backup contains plaintext data in the JSON; store it securely).

Operations Checklist

Here is a concise checklist for enabling and managing Secret encryption at rest in production.

Pre-change checklist

[ ] Verify Kubernetes version and API server configuration.
[ ] Confirm cluster-admin access and control plane node access (if self-managed).
[ ] Back up API server manifest and encryption config file (if exists).
[ ] Take etcd snapshot or backup of all Secrets.
[ ] Generate a new 32-byte random key and base64-encode it securely.
[ ] Create EncryptionConfiguration file with aescbc (or chosen provider) and identity as secondary.
[ ] Copy file to all control plane nodes with correct permissions (600).
[ ] Test in a staging cluster.

Change execution checklist

[ ] Update API server manifest on all control plane nodes to add --encryption-provider-config and volume mount.
[ ] Monitor API server pods for successful restart.
[ ] Verify API server logs show no encryption errors.
[ ] Create a test Secret and confirm it is encrypted in etcd.
[ ] Execute kubectl get secrets --all-namespaces -o json | kubectl replace -f - to rewrite all Secrets.
[ ] Verify a sample of existing Secrets are now encrypted.

Post-change verification checklist

[ ] Check API server metrics for encryption operations.
[ ] Run a read/write test for Secrets in a test namespace.
[ ] Confirm applications that use Secrets are functioning.
[ ] Remove identity provider from config (after ensuring all Secrets encrypted).
[ ] Restart API server with updated config (without identity).
[ ] Re-verify that Secrets are readable and new Secrets are encrypted.
[ ] Document the new key ID and rotation schedule.

Ongoing maintenance

[ ] Periodically rotate encryption keys (at least every 90 days or per policy).
[ ] Keep encryption config file and keys in a secure, backed-up location.
[ ] Monitor API server performance and error logs for encryption issues.
[ ] Update runbooks for key loss, rollback, and disaster recovery.

Conclusion

Encrypting Kubernetes Secrets at rest is a critical security measure for production clusters. The process is not just about flipping a flag; it requires careful planning, execution, and verification to avoid data loss or downtime.

This checklist has walked through environment inventory, safe configuration, verification, failure modes, and a step-by-step operations checklist. By following these steps, you can ensure that your Secrets are protected from unauthorized access to etcd and its backups.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Start with a low-risk verification: create a test Secret, enable encryption in a staging cluster, and practice the rollback. Then proceed to production with confidence.

Next steps: implement the checklist in your environment, integrate key rotation into your key management policy, and consider using a KMS provider for enhanced security and external key management.