## Intro

Kubernetes Secrets let you store and manage sensitive data such as passwords, API tokens, TLS certificates, and registry credentials separately from application code. This article gives you a practical, command-first guide to the basic Kubernetes Secrets operations you will use in daily work. Instead of just listing flags, each section shows a realistic command, the expected output, common failure signals, and the recovery action.

We will cover environment verification, creating secrets from literals and files, inspecting secrets without leaking them, consuming secrets in pods as environment variables and mounted files, updating and rotating secrets, troubleshooting failed mounts, and applying operational best practices.

Prerequisites: a running Kubernetes cluster, kubectl configured with a valid kubeconfig, and basic familiarity with pods and deployments. The examples use Kubernetes 1.24 syntax, but the commands work from 1.19 onward. If you are on a managed cluster, replace local paths as needed and check your RBAC permissions.

## Verify Your Environment and RBAC Before Touching Secrets

Before you create or modify a secret, confirm which cluster and namespace you are targeting. A common incident is applying a secret to the wrong namespace or cluster.

Run:

kubectl config current-context
kubectl cluster-info
kubectl version --short 
 Expected output for the current context looks like:

gke_my-project_us-central1_cluster-1
Kubernetes control plane is running at https://34.123.45.67
Client Version: v1.24.0
Server Version: v1.24.0 
 Check your permissions for secrets in the target namespace:

kubectl auth can-i create secrets --namespace default
kubectl auth can-i get secrets --namespace default 
 Expected output:

yes
yes 
 If you get no , ask an administrator for the minimum required RBAC role; do not use cluster-admin for daily secret operations.

## Create a Secret from Literal Values

The fastest way to create a secret is with kubectl create secret generic and the --from-literal flag. Never put the secret value in the command after --from-literal if shell history is enabled; use --from-literal=password=$DB_PASSWORD after making sure the environment variable is not logged, or prefer file-based creation in scripts.

Example:

kubectl create secret generic db-credentials \
 --from-literal=username=app_user \
 --from-literal=password='S3cret!Value#2024' \
 --namespace default 
 Expected output:

secret/db-credentials created 
 Verify the secret exists and shows a data size rather than the plaintext content:

kubectl get secret db-credentials -n default 
 Output:

NAME TYPE DATA AGE
db-credentials Opaque 2 15s 
 kubectl describe secret db-credentials -n default shows only key names and byte sizes, not the values:

Name: db-credentials
Namespace: default
Labels: <none>
Annotations: <none>

Type: Opaque

Data
====
password: 19 bytes
username: 8 bytes 
 This is safe for a terminal, but kubectl get secret -o yaml will print the base64-encoded value, so avoid that in shared sessions.

## Create a Secret from a File or Env File

For longer values like TLS certificates or config files, create secrets from files. The file name becomes the secret key by default.

# Create a file containing the password
echo -n 'P@ssw0rdFile' > ./db-password.txt

kubectl create secret generic file-secret \
 --from-file=./db-password.txt \
 -n default 
 Output:

secret/file-secret created 
 The secret now has key db-password.txt . To use a different key, specify --from-file=password=./db-password.txt .

For a group of environment-style variables, use --from-env-file :

cat <<EOF > ./app.env
API_KEY=abcd1234
TOKEN=token-9876
EOF

kubectl create secret generic env-secret \
 --from-env-file=./app.env \
 -n default 
 Now env-secret contains keys API_KEY and TOKEN .

## Create a Secret from a YAML Manifest

Declarative creation is preferred for version control and repeatability. Here is a generic secret manifest:

apiVersion: v1
kind: Secret
metadata:
 name: app-secret
 namespace: default
type: Opaque
data:
 username: YXBwX3VzZXI=
 password: UzNjcmV0IVZhbHVlIzIwMjQ= 
 The values in data must be base64-encoded. To generate the encoded value without shell history:

echo -n 'app_user' | base64
echo -n 'S3cret!Value#2024' | base64 
 Output:

YXBwX3VzZXI=
UzNjcmV0IVZhbHVlIzIwMjQ= 
 If you prefer plaintext in the manifest, use stringData instead of data . Kubernetes encodes it automatically on creation:

apiVersion: v1
kind: Secret
metadata:
 name: app-secret
 namespace: default
type: Opaque
stringData:
 username: app_user
 password: S3cret!Value#2024 
 Apply the manifest:

kubectl apply -f app-secret.yaml 
 Output:

secret/app-secret created 
 The secret type Opaque is used for arbitrary user-defined data. Other common types include kubernetes.io/dockerconfigjson for registry credentials and kubernetes.io/tls for TLS certs. Create those with dedicated commands:

kubectl create secret docker-registry regcred \
 --docker-server=registry.example.com \
 --docker-username=myuser \
 --docker-password=mypassword \
 --docker-email=user@example.com \
 -n default

kubectl create secret tls my-tls-secret \
 --cert=path/to/tls.crt \
 --key=path/to/tls.key \
 -n default 
 Expected output respectively:

secret/regcred created
secret/my-tls-secret created 

## Inspect Secrets Safely

 To list all secrets in a namespace:

kubectl get secrets -n default 
 Output:

NAME TYPE DATA AGE
app-secret Opaque 2 5m
db-credentials Opaque 2 10m
file-secret Opaque 1 4m
regcred kubernetes.io/dockerconfigjson 1 2m
my-tls-secret kubernetes.io/tls 2 1m 
 To see the keys contained in a secret without exposing values, use:

kubectl get secret db-credentials -n default -o jsonpath='{.data}' | jq 'keys' 
 If you do not have jq, use:

kubectl get secret db-credentials -n default -o jsonpath='{range $k, $v := .data}{$k}{"\n"}{end}' 
 Expected output:

password
username 
 To retrieve a single value and decode it locally:

kubectl get secret db-credentials -n default -o go-template='{{ .data.password | base64decode }}' 
 If your kubectl version supports base64decode function (1.18+). Otherwise:

kubectl get secret db-credentials -n default -o jsonpath='{.data.password}' | base64 -d 
 Output:

S3cret!Value#2024 
 Be careful not to redirect this to a file that may be committed to source control.

## Use Secrets in a Pod as Environment Variables

Create a deployment that injects a secret value as an environment variable. This is the simplest way for applications that read config from environment.

Example deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: webapp
 namespace: default
spec:
 replicas: 1
 selector:
 matchLabels:
 app: webapp
 template:
 metadata:
 labels:
 app: webapp
 spec:
 containers:
 - name: app
 image: nginx:1.25
 env:
 - name: DB_USER
 valueFrom:
 secretKeyRef:
 name: db-credentials
 key: username
 - name: DB_PASSWORD
 valueFrom:
 secretKeyRef:
 name: db-credentials
 key: password 
 Apply:

kubectl apply -f webapp-deployment.yaml
kubectl rollout status deployment/webapp -n default 
 Expected rollout output:

deployment "webapp" successfully rolled out 
 To verify the environment variable is set inside the container without printing the secret to logs, you can check the pod spec or use a temporary command that prints only the variable name presence:

kubectl exec deployment/webapp -- printenv DB_USER 
 Output:

app_user 
 To avoid printing the password, use:

kubectl exec deployment/webapp -- sh -c 'test -n "$DB_PASSWORD" && echo "DB_PASSWORD is set"' 
 Expected output:

DB_PASSWORD is set 
 If the secret key is missing, the pod will fail with CreateContainerConfigError . We cover troubleshooting in a later section.

## Mount a Secret as a File Volume

For applications that expect configuration files, mount the secret as a volume. Each key becomes a file in the mount path.

Example pod or deployment fragment:

apiVersion: v1
kind: Pod
metadata:
 name: secret-volume-pod
 namespace: default
spec:
 containers:
 - name: app
 image: busybox:1.36
 command: ["sleep", "3600"]
 volumeMounts:
 - name: secret-volume
 mountPath: /etc/secrets
 readOnly: true
 volumes:
 - name: secret-volume
 secret:
 secretName: db-credentials 
 Apply and exec to list the mounted files:

kubectl apply -f secret-volume-pod.yaml
kubectl exec secret-volume-pod -- ls -l /etc/secrets 
 Expected output:

total 0
lrwxrwxrwx 1 root root 13 Mar 10 10:00 password -> ..data/password
lrwxrwxrwx 1 root root 16 Mar 10 10:00 username -> ..data/username 
 The symlinks point to the current version of the secret data, allowing Kubernetes to update the mount when the secret changes (for volumes, not environment variables). You can read a file:

kubectl exec secret-volume-pod -- cat /etc/secrets/username 
 Output:

app_user 
 To mount only a specific key or set file permissions, use items and defaultMode in the volume definition:

volumes:
- name: secret-volume
 secret:
 secretName: db-credentials
 items:
 - key: username
 path: db-username
 defaultMode: 0400 
 Then the file /etc/secrets/db-username contains the username and has permissions 0400 (readable only by owner). This is useful for strict security contexts.

## Update and Rotate a Secret

Secrets are immutable in some Kubernetes versions if you set immutable: true , but the default is mutable. To update a secret without manually editing base64 values, regenerate a new manifest and apply it.

Method 1: Use kubectl create secret with --dry-run=client to produce YAML, then apply:

kubectl create secret generic db-credentials \
 --from-literal=username=app_user \
 --from-literal=password='NewP@ssword#2024' \
 --dry-run=client -o yaml | kubectl apply -f - 
 Output:

secret/db-credentials configured 
 Method 2: Edit via kubectl edit secret db-credentials -n default and manually change the base64 value. This is error-prone; use only for quick patches with immediate verification.

After updating a secret, deployments that consume it as an environment variable do not automatically restart. Trigger a rolling restart:

kubectl rollout restart deployment/webapp -n default
kubectl rollout status deployment/webapp -n default 
 For a secret mounted as a volume, the kubelet typically updates the mounted files within a minute without a restart. However, the application must watch for file changes or be restarted. For safety, perform a rollout restart after any secret rotation that affects running workloads.

## Troubleshoot Common Secret Failures

### 1. Pod fails with CreateContainerConfigError

If a pod references a secret key that does not exist, the container cannot start. Describe the pod:

kubectl describe pod webapp-7c6f6f6c9-abcde -n default 
 Look for events:

Warning Failed 16s (x2 over 20s) kubelet Error: secret "db-credentials" not found
Warning Failed 16s kubelet Error: couldn't find key username in Secret default/db-credentials 
 Fix: Create the secret or correct the key name, then delete the failed pod (if not managed by a controller) or apply a corrected manifest.

### 2. Permission denied when reading a mounted secret

If your container runs as a non-root user and the secret volume has default permissions 0644 or 0600 , but the file owner is root, the app may fail with permission denied when trying to read.

Example error in app logs:

open /etc/secrets/password: permission denied 
 Fix: Set defaultMode: 0444 in the secret volume definition, or set fsGroup in the pod security context so that the files are group-readable by the specified group:

spec:
 securityContext:
 fsGroup: 2000
 containers:
 - name: app
 ... 
 Then reapply and restart the pod.

### 3. Base64 decoding issues

When creating secrets manually from data , ensure the value is valid base64. Commands like echo add a newline, so use echo -n or printf . Example of wrong encoding:

echo 'password' | base64 # includes newline, produces cGFzc3dvcmQK 
 The trailing Cg== or newline may cause the application to receive an extra newline character. Use:

printf 'password' | base64 # cGFzc3dvcmQ= 
 If you already created the secret, inspect the decoded length:

kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d | wc -c 
 The expected byte count should match the original; if it is one byte larger, likely a trailing newline.

### 4. Secret referenced in the wrong namespace

Secrets are namespaced objects. If a pod in namespace production references db-credentials , the secret must exist in production , not default . Confirm:

kubectl get secret db-credentials -n production 
 If it does not exist, create it in the correct namespace or adjust the pod manifest to use the fully qualified name (which is not possible for secret references; they must be in the same namespace as the pod). Cross-namespace access requires copying the secret or using a controller like Kubernetes External Secrets.

## Security and Operational Best Practices for Daily Operations

- Never use kubectl get secret -o yaml in a shared or recorded terminal. The base64 values can be decoded instantly. Prefer kubectl describe secret for key names or jq for keys.

- Enable encryption at rest in your cluster's etcd. On managed clusters, this is often a setting. On self-managed, configure a KMS plugin or at least use --encryption-provider-config on the API server. This protects secrets if etcd backups leak.

- Apply least privilege RBAC. Only grant secret read access to service accounts that need it. Use kubectl create role and kubectl create rolebinding to grant access to specific secrets or namespaces. Example to allow a service account to read only the db-credentials secret:

kubectl create role secret-reader --verb=get --verb=list --resource=secrets --resource-name=db-credentials -n default
kubectl create rolebinding secret-reader-binding --role=secret-reader --serviceaccount=default:my-app-sa -n default 

- Avoid checking secrets into git. Use .gitignore for *-secret.yaml , or use templating with sealed secrets or external secrets operators for gitops workflows.

- Set immutable: true on secrets that are not expected to change. This prevents accidental updates and improves performance. Example:

 apiVersion: v1
kind: Secret
metadata:
 name: static-secret
immutable: true
data:
 key: dmFsdWU= 

- Regularly rotate secrets and test that applications pick up new values after a restart. Use a canary deployment or a pod that reads the secret and verifies length.

## Operations Checklist for Daily Secret Management

 Use this ordered checklist when working with Kubernetes Secrets:

- [ ] Confirm current context and namespace: kubectl config current-context && kubectl config get-contexts

- [ ] Verify RBAC: kubectl auth can-i create secrets -n <namespace>

- [ ] Capture current state: kubectl get secrets -n <namespace> -o name

- [ ] Create or update the secret from a file or manifest, never from shell history if sensitive

- [ ] Verify creation: kubectl get secret <name> -n <namespace> and kubectl describe secret <name> -n <namespace>

- [ ] Decode a sample value to confirm it matches expected plaintext (in a secure shell)

- [ ] Apply pod or deployment manifest that consumes the secret

- [ ] Check rollout status: kubectl rollout status deployment/<name> -n <namespace>

- [ ] Test access inside the pod without printing secrets: kubectl exec -- sh -c 'test -n "$VAR" && echo set'

- [ ] For volume mounts, verify file permissions and content: kubectl exec -- ls -l /path

- [ ] Set immutable: true if the secret should not change

- [ ] Document the secret owner, rotation interval, and recovery procedure

## Conclusion

 Kubernetes Secrets basic commands become useful when they are combined with safe habits: verify the target cluster, create secrets from files or manifests, inspect them without exposing values, consume them explicitly in pods, and test the result. This article covered the core operations from creation to troubleshooting, with concrete commands and expected outputs.

As a next step, choose one low-risk secret in a development namespace and run through the full workflow: create it from a manifest, mount it as a file, verify permissions, then simulate a missing key to see the failure mode. Practice a rotation with a rolling restart and observe how the application behaves. That hands-on exercise will turn these commands into a reliable daily routine.