## Intro

Kubernetes Secrets hold credentials, API keys, tokens, TLS material, and other critical values. Backing them up and restoring them safely is table-stakes for disaster recovery, but is often left implicit until an outage forces action. This practical guide shows how to:

- Inventory and scope a safe pilot for Secrets backup

- Export Secrets with minimal, reusable manifests

- Encrypt and store backups

- Restore Secrets cleanly to a namespace or a new cluster

- Verify correctness with concrete, observable checks

- Recognize failure modes and recover predictably

- Run with a repeatable operations checklist

The examples assume a Unix-like shell and kubectl access. All examples are constructed and use hypothetical names and values for clarity.

## Version and Environment Inventory

Before touching anything, record your environment. This helps you pick the right commands, avoid surprises, and keep a trail you can repeat later.

Prerequisites:

- Kubernetes: v1.19+ recommended (supports immutable Secrets). Any maintained version works.

- kubectl : Match your cluster minor version when possible.

- Access: You need at least get/list/watch/create/patch/delete on Secrets in the target namespaces. For full-cluster backup, you need to list Secrets cluster-wide.

- Optional tools to simplify export and conversion: jq , yq , openssl , sops / age (for encryption-at-rest of backups).

- A narrow pilot scope: one namespace and 1-3 critical Secrets. Keep it measurable and locally inspectable.

Quick inventory commands (read-only):

kubectl version --short
kubectl config get-contexts
kubectl auth can-i list secrets -A
kubectl get ns 
 Record the target namespace(s) and Secrets you will pilot. For example: demo namespace, secrets ai-api , app-tls .

## Safe Configuration Path

This section lays out a scoped, repeatable path you can adopt today. It focuses on kubectl -based export with minimal metadata, optional encryption at rest, and straightforward restores. You can extend it later with your preferred key management or automation.

### Decide what to back up and what to skip

Not every Secret belongs in your backup set. Start with high-value, low-churn items.

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Secret type or usage</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Include in backup?</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Notes</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Opaque (API keys, DB creds)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Yes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Core application credentials.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">kubernetes.io/tls</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Yes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Certificates and keys; verify expiry on restore.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">kubernetes.io/dockerconfigjson</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Yes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Required for image pulls in private registries.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Service account tokens (legacy)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Usually no</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Ephemeral in modern clusters (1.24+). Recreate via service accounts if needed.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Generated or short-lived tokens</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Usually no</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Prefer regeneration procedures over backup.</td></tr></tbody>
</table>
</div>

### Identify target Secrets (pilot)

Constructed example commands for a pilot in the demo namespace:

# List Secrets and types
kubectl get secrets -n demo -o custom-columns=NAME:.metadata.name,TYPE:.type

# Filter likely user-managed Secrets (excluding legacy service account tokens)
kubectl get secrets -n demo -o json | jq -r '.items[] | select(.type!="kubernetes.io/service-account-token\)) | [.metadata.name,.type] | @tsv' 
 Pick 1-3 Secrets for the pilot, for example:

- ai-api (Opaque) with an API key

- app-tls (kubernetes.io/tls) for inbound HTTPS

### Export a single Secret with minimal metadata

kubectl returns base64-encoded data plus many metadata fields you do not want in a reusable backup. Remove volatile fields so the manifest can be applied cleanly elsewhere.

Constructed example: export ai-api Secret from demo as minimal YAML:

- Export to JSON and strip volatile metadata:

kubectl get secret ai-api -n demo -o json \
 | jq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields, .metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"], .metadata.generation)' \
 > ai-api.json 

- Optionally convert JSON to YAML for readability:

 yq -P e ai-api.json > ai-api.yaml 
 The resulting ai-api.yaml should look similar to this constructed example (values are hypotheticals):

apiVersion: v1
kind: Secret
metadata:
 name: ai-api
 namespace: demo
type: Opaque
data:
 OPENAI_API_KEY: c2stdGVzdC1BQUFCQ0QxMjM0NTY= 
 Caution: data values are base64-encoded, not encrypted. Treat these files as sensitive and restrict access.

### Export TLS Secrets

For TLS, the type should be kubernetes.io/tls with two keys: tls.crt and tls.key .

kubectl get secret app-tls -n demo -o json \
 | jq 'del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields, .metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"], .metadata.generation)' \
 > app-tls.json
yq -P e app-tls.json > app-tls.yaml 

### Bulk export for a namespace

 For a simple, per-Secret file structure (easier diffing and restore):

NS=demo
TS=$(date +%Y%m%d-%H%M%S)
mkdir -p backups/$NS-$TS
for s in $(kubectl get secrets -n $NS -o jsonpath='{.items[*].metadata.name}'); do
 kubectl get secret "$s" -n $NS -o json \
 | jq 'select(.type != "kubernetes.io/service-account-token\)) | del(.metadata.uid, .metadata.resourceVersion, .metadata.creationTimestamp, .metadata.managedFields, .metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"], .metadata.generation)' \
 > backups/$NS-$TS/$s.json
 yq -P e backups/$NS-$TS/$s.json > backups/$NS-$TS/$s.yaml
 rm backups/$NS-$TS/$s.json
done 
 Adjust filters per your environment. Keep both JSON and YAML if you prefer; YAML is shown for readability.

### Optional: Encrypt backups at rest

If you store backups outside a dedicated vault, use file-level encryption. As one option, encrypt with sops and age (constructed example):

# Generate an age key pair (once)
age-keygen -o ~/.age/key.txt

# Encrypt a YAML file for your public recipient
sops --encrypt --age YOUR_AGE_PUBLIC_KEY backups/demo-20260101-000000/ai-api.yaml \
 > backups/demo-20260101-000000/ai-api.enc.yaml

# Decrypt when restoring
sops --decrypt backups/demo-20260101-000000/ai-api.enc.yaml > /tmp/ai-api.yaml 
 Restrict filesystem permissions for keys and backups (for example, chmod 600 on sensitive files and directories).

### Restore a single Secret (same cluster or new cluster)

- Ensure the target namespace exists:

kubectl get ns demo || kubectl create ns demo 

- Apply the Secret manifest:

 kubectl apply -f ai-api.yaml 

- If a Secret was previously marked immutable: true and changed values are required, delete then recreate:

 # Check if immutable
kubectl get secret ai-api -n demo -o jsonpath='{.immutable}' 2>/dev/null || true

# If immutable is true, do a delete-then-apply:
kubectl delete secret ai-api -n demo
kubectl apply -f ai-api.yaml 

- For TLS, apply similarly:

 kubectl apply -f app-tls.yaml 

- Restart workloads that consume environment variables from Secrets (pods do not automatically reload env vars). For deployments:

 kubectl rollout restart deploy/myapp -n demo 
 Volume mounts reflect Secret updates shortly after modification; env vars require a pod restart.

### Restore multiple Secrets

If you exported per-Secret YAML files into a directory:

kubectl apply -f backups/demo-20260101-000000/ 
 If files include immutable Secrets and values actually changed, pre-delete as needed (or remove immutable: true from the file, delete, then apply).

## Verification and Diagnostics

You need clear, observable checks that prove both backup artifacts and restored objects are correct.

### Verify backup files locally

# Inspect metadata and type
yq e '.kind, .metadata.name, .metadata.namespace, .type' ai-api.yaml

# Verify expected keys exist (constructed example expects OPENAI_API_KEY)
yq e '.data | keys' ai-api.yaml

# Optionally decode and compare values offline (store outputs securely)
yq -r '.data.OPENAI_API_KEY' ai-api.yaml | base64 -d | sha256sum 
 Record hashes for comparison after restore.

### Verify restored Secret in cluster

# Confirm Secret presence and type
kubectl get secret ai-api -n demo -o jsonpath='{.type}{"\n"}'

# Compare key set and hashes to the backup
kubectl get secret ai-api -n demo -o json \
 | jq -r '.data | to_entries[] | "\(.key) \(.value)"}' \
 | while read k v; do echo -n "$k " ; echo -n "$v" | base64 -d | sha256sum; done

# For TLS, decode and check CN and expiry (constructed example)
kubectl get secret app-tls -n demo -o jsonpath='{.data.tls\.crt}' \
 | base64 -d > /tmp/tls.crt
openssl x509 -in /tmp/tls.crt -noout -subject -dates -fingerprint -sha256 
 Expected results:

- Types match (Opaque vs kubernetes.io/tls)

- Key sets match

- Hashes match previous values (if unchanged)

- TLS cert has expected subject and valid dates

### Verify workloads consume Secrets

# Pods mounting Secrets as volumes: check the files exist and have expected content and permissions
kubectl exec -n demo deploy/myapp -- sh -c 'ls -l /etc/secrets && head -c 16 /etc/secrets/some-key | wc -c'

# Pods using env vars from Secrets: confirm value presence (avoid printing secrets). Use indirect checks or application health endpoints.

# Rollout status is healthy after restarts
kubectl rollout status deploy/myapp -n demo --timeout=120s 

### Diagnostics for common issues

 # Permission issue
kubectl auth can-i create secrets -n demo

# Namespace mismatch
kubectl get secret ai-api -A | grep ai-api

# Wrong context
kubectl config current-context

# Events referencing Secrets
kubectl get events -n demo --sort-by=.lastTimestamp | grep -i secret || true 

## Failure Modes and Recovery

 Plan for the ways Secret backups and restores can fail, so you can recover without scrambling.

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Failure mode</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Symptom</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Recovery</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Missing namespace</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl apply</code> errors: namespace not found</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Create namespace, re-apply.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Immutable Secret changed</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Apply fails; immutable field prevents update</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Delete Secret, then apply new manifest.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Type mismatch on restore</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Apply may succeed, but workloads expect a specific type</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Ensure <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">.type</code> matches original; delete-and-recreate if needed.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Corrupted or non-base64 data</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl apply</code> or pod mount fails</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Recreate file from a known-good backup; verify base64 and keys.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Incomplete export (metadata-only)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Secret exists but lacks keys</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Re-export ensuring <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">.data</code> keys present; verify before restore.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Wrong cluster/context</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Secrets restored to unintended cluster</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Verify <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl</code> context before apply; label backups with cluster ID.</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Workloads not reloading env vars</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">App still sees old values</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Rollout restart deployments/statefulsets using the Secret.</td></tr></tbody>
</table>
</div>
Additional risks and mitigations:

- Storage leakage: Backups contain sensitive data. Use encryption at rest and strict file permissions.

- Over-broad backups: Start with a pilot namespace to reduce blast radius, then incrementally expand.

- TLS expiry drift: On restore, confirm cert validity window to avoid immediate outages.

### Disaster recovery scenarios

- Restore to a new cluster after loss

- Recreate namespaces.

- Apply Secret manifests (start with core dependencies: image pull Secrets, app credentials, TLS certs).

- Restore ConfigMaps and workloads.

- Trigger rollouts and verify end-to-end health.

- Roll back a Secret value to a previous known-good version

- Identify the timestamped backup file.

- If immutable: delete current Secret.

- Apply the older manifest.

- Restart affected workloads (env var consumers) and verify.

Identify missing Secrets using a manifest inventory vs live cluster comparison:

- Partial namespace loss

comm -23 <(ls backups/demo-20260101-000000 | sed 's/.yaml$//' | sort) <(kubectl get secrets -n demo -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n' | sort) 

- Apply only the missing Secrets.

## Operations Checklist

 Use this as a repeatable runbook for pilot and production.

Daily or per-change (as appropriate):

- Confirm kubectl context and target namespace(s).

- Export changed Secrets with minimal metadata.

- Encrypt and store backups with restricted permissions.

Weekly:

- Perform a test restore in a non-production namespace.

- Verify hashes and, for TLS, certificate validity.

- Confirm workloads reload as expected (volume vs env var semantics).

Monthly or quarterly:

- Audit RBAC for who can read and back up Secrets.

- Rotate encryption keys for backup-at-rest if policy requires.

- Review inventory and add/remove Secrets from backup scope.

Before restore in production:

- Validate target namespace exists and is correct.

- Confirm the manifest matches intended .type and keys.

- For immutable Secrets, plan delete-then-apply window and workload restart.

After restore:

- Verify Secret presence, type, and data hashes.

- Restart env var consumers and confirm rollout success.

- Check application health endpoints and logs for credential or TLS errors.

- Document what was restored, when, and by whom.

## Practical examples

### Constructed example 1: Back up and restore an API key Secret

- Create and back up:

- Secret name: ai-api , namespace: demo , type: Opaque

- Export minimal YAML as shown earlier to ai-api.yaml

- Encrypt to ai-api.enc.yaml (optional)

- Restore:

- Decrypt if needed: sops --decrypt ai-api.enc.yaml > /tmp/ai-api.yaml

- kubectl apply -f /tmp/ai-api.yaml

- Verify hash matches backup

- Restart deployment using it: kubectl rollout restart deploy/myapp -n demo

### Constructed example 2: Back up and restore a TLS Secret

- Back up app-tls :

- Export minimal YAML to app-tls.yaml

- Optionally store the decoded tls.crt separately for quick openssl checks (store securely)

- Restore:

- kubectl apply -f app-tls.yaml

- Verify with openssl x509 -noout -subject -dates -fingerprint -sha256

- Confirm ingress or service resumes healthy TLS handshakes

## Conclusion

A reliable Kubernetes Secrets backup and restore procedure is small, testable, and observable. Start with a narrow pilot in one namespace, export minimal manifests, encrypt them at rest, and practice restores with explicit verification steps. As you expand beyond the pilot, codify the checklist, label and version your backups, and rehearse disaster scenarios. When restore is a clear and distinct operation with predictable verification, you reduce rework and shorten recovery times. Your next step: choose one namespace, pick two Secrets, and run a full backup-verify-restore drill end-to-end today.