Kubernetes Secrets: Basic Commands and Practical Examples for Daily Operations

Intro

Kubernetes Secrets let you store and manage sensitive data such as passwords, API tokens, TLS certificates, and registry credentials separately from application code. This article gives you a practical, command-first guide to the basic Kubernetes Secrets operations you will use in daily work. Instead of just listing flags, each section shows a realistic command, the expected output, common failure signals, and the recovery action.

We will cover environment verification, creating secrets from literals and files, inspecting secrets without leaking them, consuming secrets in pods as environment variables and mounted files, updating and rotating secrets, troubleshooting failed mounts, and applying operational best practices.

Prerequisites: a running Kubernetes cluster, kubectl configured with a valid kubeconfig, and basic familiarity with pods and deployments. The examples use Kubernetes 1.24 syntax, but the commands work from 1.19 onward. If you are on a managed cluster, replace local paths as needed and check your RBAC permissions.

Verify Your Environment and RBAC Before Touching Secrets

Before you create or modify a secret, confirm which cluster and namespace you are targeting. A common incident is applying a secret to the wrong namespace or cluster.

Run:

kubectl config current-context
kubectl cluster-info
kubectl version --short

Expected output for the current context looks like:

gke_my-project_us-central1_cluster-1
Kubernetes control plane is running at https://34.123.45.67
Client Version: v1.24.0
Server Version: v1.24.0

Check your permissions for secrets in the target namespace:

kubectl auth can-i create secrets --namespace default
kubectl auth can-i get secrets --namespace default

Expected output:

yes
yes

If you get no, ask an administrator for the minimum required RBAC role; do not use cluster-admin for daily secret operations.

Create a Secret from Literal Values

The fastest way to create a secret is with kubectl create secret generic and the --from-literal flag. Never put the secret value in the command after --from-literal if shell history is enabled; use --from-literal=password=$DB_PASSWORD after making sure the environment variable is not logged, or prefer file-based creation in scripts.

Example:

kubectl create secret generic db-credentials \
  --from-literal=username=app_user \
  --from-literal=password='S3cret!Value#2024' \
  --namespace default

Expected output:

secret/db-credentials created

Verify the secret exists and shows a data size rather than the plaintext content:

kubectl get secret db-credentials -n default

Output:

NAME             TYPE     DATA   AGE
db-credentials   Opaque   2      15s

kubectl describe secret db-credentials -n default shows only key names and byte sizes, not the values:

Name:         db-credentials
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
password:  19 bytes
username:  8 bytes

This is safe for a terminal, but kubectl get secret -o yaml will print the base64-encoded value, so avoid that in shared sessions.

Create a Secret from a File or Env File

For longer values like TLS certificates or config files, create secrets from files. The file name becomes the secret key by default.

# Create a file containing the password
echo -n 'P@ssw0rdFile' > ./db-password.txt

kubectl create secret generic file-secret \
  --from-file=./db-password.txt \
  -n default

Output:

secret/file-secret created

The secret now has key db-password.txt. To use a different key, specify --from-file=password=./db-password.txt.

For a group of environment-style variables, use --from-env-file:

cat <<EOF > ./app.env
API_KEY=abcd1234
TOKEN=token-9876
EOF

kubectl create secret generic env-secret \
  --from-env-file=./app.env \
  -n default

Now env-secret contains keys API_KEY and TOKEN.

Create a Secret from a YAML Manifest

Declarative creation is preferred for version control and repeatability. Here is a generic secret manifest:

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
  namespace: default
type: Opaque
data:
  username: YXBwX3VzZXI=
  password: UzNjcmV0IVZhbHVlIzIwMjQ=

The values in data must be base64-encoded. To generate the encoded value without shell history:

echo -n 'app_user' | base64
echo -n 'S3cret!Value#2024' | base64

Output:

YXBwX3VzZXI=
UzNjcmV0IVZhbHVlIzIwMjQ=

If you prefer plaintext in the manifest, use stringData instead of data. Kubernetes encodes it automatically on creation:

apiVersion: v1
kind: Secret
metadata:
  name: app-secret
  namespace: default
type: Opaque
stringData:
  username: app_user
  password: S3cret!Value#2024

Apply the manifest:

kubectl apply -f app-secret.yaml

Output:

secret/app-secret created

The secret type Opaque is used for arbitrary user-defined data. Other common types include kubernetes.io/dockerconfigjson for registry credentials and kubernetes.io/tls for TLS certs. Create those with dedicated commands:

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=myuser \
  --docker-password=mypassword \
  [email protected] \
  -n default

kubectl create secret tls my-tls-secret \
  --cert=path/to/tls.crt \
  --key=path/to/tls.key \
  -n default

Expected output respectively:

secret/regcred created
secret/my-tls-secret created

Inspect Secrets Safely

To list all secrets in a namespace:

kubectl get secrets -n default

Output:

NAME             TYPE                             DATA   AGE
app-secret       Opaque                           2      5m
db-credentials   Opaque                           2      10m
file-secret      Opaque                           1      4m
regcred          kubernetes.io/dockerconfigjson   1      2m
my-tls-secret    kubernetes.io/tls                2      1m

To see the keys contained in a secret without exposing values, use:

kubectl get secret db-credentials -n default -o jsonpath='{.data}' | jq 'keys'

If you do not have jq, use:

kubectl get secret db-credentials -n default -o jsonpath='{range $k, $v := .data}{$k}{"\n"}{end}'

Expected output:

password
username

To retrieve a single value and decode it locally:

kubectl get secret db-credentials -n default -o go-template='{{ .data.password | base64decode }}'

If your kubectl version supports base64decode function (1.18+). Otherwise:

kubectl get secret db-credentials -n default -o jsonpath='{.data.password}' | base64 -d

Output:

S3cret!Value#2024

Be careful not to redirect this to a file that may be committed to source control.

Use Secrets in a Pod as Environment Variables

Create a deployment that injects a secret value as an environment variable. This is the simplest way for applications that read config from environment.

Example deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: app
        image: nginx:1.25
        env:
        - name: DB_USER
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: username
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: password

Apply:

kubectl apply -f webapp-deployment.yaml
kubectl rollout status deployment/webapp -n default

Expected rollout output:

deployment "webapp" successfully rolled out

To verify the environment variable is set inside the container without printing the secret to logs, you can check the pod spec or use a temporary command that prints only the variable name presence:

kubectl exec deployment/webapp -- printenv DB_USER

Output:

app_user

To avoid printing the password, use:

kubectl exec deployment/webapp -- sh -c 'test -n "$DB_PASSWORD" && echo "DB_PASSWORD is set"'

Expected output:

DB_PASSWORD is set

If the secret key is missing, the pod will fail with CreateContainerConfigError. We cover troubleshooting in a later section.

Mount a Secret as a File Volume

For applications that expect configuration files, mount the secret as a volume. Each key becomes a file in the mount path.

Example pod or deployment fragment:

apiVersion: v1
kind: Pod
metadata:
  name: secret-volume-pod
  namespace: default
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep", "3600"]
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: db-credentials

Apply and exec to list the mounted files:

kubectl apply -f secret-volume-pod.yaml
kubectl exec secret-volume-pod -- ls -l /etc/secrets

Expected output:

total 0
lrwxrwxrwx    1 root     root            13 Mar 10 10:00 password -> ..data/password
lrwxrwxrwx    1 root     root            16 Mar 10 10:00 username -> ..data/username

The symlinks point to the current version of the secret data, allowing Kubernetes to update the mount when the secret changes (for volumes, not environment variables). You can read a file:

kubectl exec secret-volume-pod -- cat /etc/secrets/username

Output:

app_user

To mount only a specific key or set file permissions, use items and defaultMode in the volume definition:

volumes:
- name: secret-volume
  secret:
    secretName: db-credentials
    items:
    - key: username
      path: db-username
    defaultMode: 0400

Then the file /etc/secrets/db-username contains the username and has permissions 0400 (readable only by owner). This is useful for strict security contexts.

Update and Rotate a Secret

Secrets are immutable in some Kubernetes versions if you set immutable: true, but the default is mutable. To update a secret without manually editing base64 values, regenerate a new manifest and apply it.

Method 1: Use kubectl create secret with --dry-run=client to produce YAML, then apply:

kubectl create secret generic db-credentials \
  --from-literal=username=app_user \
  --from-literal=password='NewP@ssword#2024' \
  --dry-run=client -o yaml | kubectl apply -f -

Output:

secret/db-credentials configured

Method 2: Edit via kubectl edit secret db-credentials -n default and manually change the base64 value. This is error-prone; use only for quick patches with immediate verification.

After updating a secret, deployments that consume it as an environment variable do not automatically restart. Trigger a rolling restart:

kubectl rollout restart deployment/webapp -n default
kubectl rollout status deployment/webapp -n default

For a secret mounted as a volume, the kubelet typically updates the mounted files within a minute without a restart. However, the application must watch for file changes or be restarted. For safety, perform a rollout restart after any secret rotation that affects running workloads.

Troubleshoot Common Secret Failures

1. Pod fails with CreateContainerConfigError

If a pod references a secret key that does not exist, the container cannot start. Describe the pod:

kubectl describe pod webapp-7c6f6f6c9-abcde -n default

Look for events:

Warning  Failed     16s (x2 over 20s)  kubelet  Error: secret "db-credentials" not found
Warning  Failed     16s                kubelet  Error: couldn't find key username in Secret default/db-credentials

Fix: Create the secret or correct the key name, then delete the failed pod (if not managed by a controller) or apply a corrected manifest.

2. Permission denied when reading a mounted secret

If your container runs as a non-root user and the secret volume has default permissions 0644 or 0600, but the file owner is root, the app may fail with permission denied when trying to read.

Example error in app logs:

open /etc/secrets/password: permission denied

Fix: Set defaultMode: 0444 in the secret volume definition, or set fsGroup in the pod security context so that the files are group-readable by the specified group:

spec:
  securityContext:
    fsGroup: 2000
  containers:
  - name: app
    ...

Then reapply and restart the pod.

3. Base64 decoding issues

When creating secrets manually from data, ensure the value is valid base64. Commands like echo add a newline, so use echo -n or printf. Example of wrong encoding:

echo 'password' | base64  # includes newline, produces cGFzc3dvcmQK

The trailing Cg== or newline may cause the application to receive an extra newline character. Use:

printf 'password' | base64  # cGFzc3dvcmQ=

If you already created the secret, inspect the decoded length:

kubectl get secret my-secret -o jsonpath='{.data.password}' | base64 -d | wc -c

The expected byte count should match the original; if it is one byte larger, likely a trailing newline.

4. Secret referenced in the wrong namespace

Secrets are namespaced objects. If a pod in namespace production references db-credentials, the secret must exist in production, not default. Confirm:

kubectl get secret db-credentials -n production

If it does not exist, create it in the correct namespace or adjust the pod manifest to use the fully qualified name (which is not possible for secret references; they must be in the same namespace as the pod). Cross-namespace access requires copying the secret or using a controller like Kubernetes External Secrets.

Security and Operational Best Practices for Daily Operations

1. Never use kubectl get secret -o yaml in a shared or recorded terminal. The base64 values can be decoded instantly. Prefer kubectl describe secret for key names or jq for keys.

1. Enable encryption at rest in your cluster's etcd. On managed clusters, this is often a setting. On self-managed, configure a KMS plugin or at least use --encryption-provider-config on the API server. This protects secrets if etcd backups leak.

1. Apply least privilege RBAC. Only grant secret read access to service accounts that need it. Use kubectl create role and kubectl create rolebinding to grant access to specific secrets or namespaces. Example to allow a service account to read only the db-credentials secret:

kubectl create role secret-reader --verb=get --verb=list --resource=secrets --resource-name=db-credentials -n default
kubectl create rolebinding secret-reader-binding --role=secret-reader --serviceaccount=default:my-app-sa -n default

1. Avoid checking secrets into git. Use .gitignore for *-secret.yaml, or use templating with sealed secrets or external secrets operators for gitops workflows.

1. Set immutable: true on secrets that are not expected to change. This prevents accidental updates and improves performance. Example:

apiVersion: v1
kind: Secret
metadata:
  name: static-secret
immutable: true
data:
  key: dmFsdWU=

1. Regularly rotate secrets and test that applications pick up new values after a restart. Use a canary deployment or a pod that reads the secret and verifies length.

Operations Checklist for Daily Secret Management

Use this ordered checklist when working with Kubernetes Secrets:

• [ ] Confirm current context and namespace: kubectl config current-context && kubectl config get-contexts
• [ ] Verify RBAC: kubectl auth can-i create secrets -n <namespace>
• [ ] Capture current state: kubectl get secrets -n <namespace> -o name
• [ ] Create or update the secret from a file or manifest, never from shell history if sensitive
• [ ] Verify creation: kubectl get secret <name> -n <namespace> and kubectl describe secret <name> -n <namespace>
• [ ] Decode a sample value to confirm it matches expected plaintext (in a secure shell)
• [ ] Apply pod or deployment manifest that consumes the secret
• [ ] Check rollout status: kubectl rollout status deployment/<name> -n <namespace>
• [ ] Test access inside the pod without printing secrets: kubectl exec <pod> -- sh -c 'test -n "$VAR" && echo set'
• [ ] For volume mounts, verify file permissions and content: kubectl exec <pod> -- ls -l /path
• [ ] Set immutable: true if the secret should not change
• [ ] Document the secret owner, rotation interval, and recovery procedure

Conclusion

Kubernetes Secrets basic commands become useful when they are combined with safe habits: verify the target cluster, create secrets from files or manifests, inspect them without exposing values, consume them explicitly in pods, and test the result. This article covered the core operations from creation to troubleshooting, with concrete commands and expected outputs.

As a next step, choose one low-risk secret in a development namespace and run through the full workflow: create it from a manifest, mount it as a file, verify permissions, then simulate a missing key to see the failure mode. Practice a rotation with a rolling restart and observe how the application behaves. That hands-on exercise will turn these commands into a reliable daily routine.