Kubernetes Secret Basic Commands with Practical Examples

Intro

Secrets in Kubernetes are objects that store sensitive information such as passwords, OAuth tokens, and SSH keys. Using Secrets instead of plain text in Pod specs or container images reduces the risk of accidental exposure and makes credential rotation more manageable. However, many engineers only memorize a few kubectl commands and do not practice the observation, verification, and recovery steps that turn those commands into reliable operations.

This article covers the core kubectl commands for creating, inspecting, mounting, and troubleshooting Secrets. It is written for developers, DevOps consultants, and startup teams who need practical examples with clear expected output and failure signals. Each section includes commands, sample outputs, and what to check when something does not behave as expected. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of real secrets, and verify every change.

You need a working Kubernetes cluster and kubectl configured to follow along. The examples assume a namespace named demo; create it with kubectl create namespace demo if it does not exist. Some commands use base64 for encoding, available on macOS, Linux, and Windows Subsystem for Linux.

Version and Environment Inventory

Before running any Secret commands, confirm your cluster version and API availability. Kubernetes versions 1.19 and later include the stable v1 Secret API, but older clusters may require v1beta1. The kubectl version command shows both client and server versions:

kubectl version --short

Sample output:

Client Version: v1.28.2
Server Version: v1.27.3

If the server version is older than 1.19, upgrade or use the appropriate API version in your manifests. Check that you can read Secrets in the target namespace with a non-destructive command:

kubectl auth can-i list secrets --namespace demo

Expected output when permission exists:

yes

If it returns no, ask your cluster administrator to grant read access via a Role or ClusterRole. Also verify the default ServiceAccount token is not being used for secret access if you have disabled auto-mounting:

kubectl get serviceaccount default -o yaml --namespace demo | grep automountServiceAccountToken

If nothing is returned, auto-mounting is enabled by default and may be a security consideration. A read-only environment inventory ensures that you do not attempt changes your role cannot support.

Creating Secrets

Kubernetes supports several ways to create Secrets: from literal values, from files, and from a YAML manifest. The kubectl create secret generic command is quick for simple cases, but it writes the secret value into your shell history. Use it only for development; in production, prefer declarative manifests applied via CI/CD where the manifest source is protected.

From Literal Values

Create a Secret with username and password literals:

kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password='S3cret!' \
  --namespace demo

Expected output:

secret/db-credentials created

Verify the Secret was created:

kubectl get secret db-credentials --namespace demo

Output:

NAME             TYPE     DATA   AGE
db-credentials   Opaque   2      10s

From Files

If you already have credential files, create a Secret from them:

echo -n 'admin' > username.txt
echo -n 'S3cret!' > password.txt
kubectl create secret generic db-credentials-file \
  --from-file=./username.txt \
  --from-file=./password.txt \
  --namespace demo

The keys will be the filenames (username.txt, password.txt). To specify custom keys, use --from-file=username=./username.txt --from-file=password=./password.txt.

Inspect the resulting data keys:

kubectl get secret db-credentials-file -o jsonpath='{.data}' --namespace demo

Output (base64 encoded):

{"password.txt":"UzNjcmV0IQ==","username.txt":"YWRtaW4="}

From a Manifest

For production, write a YAML file with base64-encoded values:

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials-manifest
  namespace: demo
type: Opaque
data:
  username: YWRtaW4=  # admin
  password: UzNjcmV0IQ==  # S3cret!

Apply it:

kubectl apply -f db-secret.yaml

Expected output:

secret/db-credentials-manifest created

If the Secret already exists, apply updates it (subject to validation). You can generate the base64 values with:

echo -n 'admin' | base64
# YWRtaW4=
echo -n 'S3cret!' | base64
# UzNjcmV0IQ==

Never commit raw manifest files with real base64 values to a public repository; use sealed secrets or an external secret manager for encryption.

Reading and Decoding Secrets

To see what is inside a Secret without decoding manually, use kubectl get secret -o yaml and then decode the base64 values:

kubectl get secret db-credentials-manifest -o yaml --namespace demo

Output snippet:

data:
  password: UzNjcmV0IQ==
  username: YWRtaW4=
kind: Secret
metadata:
  ...

Decode a value:

echo 'UzNjcmV0IQ==' | base64 --decode
# S3cret!

For quick extraction of a single value:

kubectl get secret db-credentials-manifest \
  -o jsonpath='{.data.password}' --namespace demo | base64 --decode

Caution: Decoding secrets on your workstation exposes them in terminal output. Limit this to troubleshooting and avoid piping to logs or shared screens.

Mounting Secrets as Environment Variables

A Pod can consume Secrets as environment variables. This is common for application configuration. Create a Pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
  namespace: demo
spec:
  containers:
  - name: app
    image: busybox:1.28
    command: ["sleep", "3600"]
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASS
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

Apply and verify:

kubectl apply -f secret-env-pod.yaml
kubectl exec secret-env-pod -- namespace demo -- printenv DB_USER

Expected output:

admin

If the environment variable is missing, check the Pod events for invalid Secret references:

kubectl describe pod secret-env-pod --namespace demo

Look for events like Error: secret "db-credentials" not found.

Mounting Secrets as Volumes

Files provide a dynamic way to consume Secrets; changes to the Secret are reflected in the mounted volume (though the application must reload the file). Example Pod with a volume mount:

apiVersion: v1
kind: Pod
metadata:
  name: secret-volume-pod
  namespace: demo
spec:
  containers:
  - name: app
    image: busybox:1.28
    command: ["sleep", "3600"]
    volumeMounts:
    - name: credentials
      mountPath: /etc/credentials
      readOnly: true
  volumes:
  - name: credentials
    secret:
      secretName: db-credentials

After applying, check the files:

kubectl exec secret-volume-pod --namespace demo -- ls /etc/credentials
# password  username
kubectl exec secret-volume-pod --namespace demo -- cat /etc/credentials/username
# admin

If the mount fails, the Pod will not start. Check the Pod status and events:

kubectl get pod secret-volume-pod --namespace demo
kubectl describe pod secret-volume-pod --namespace demo

Common errors include missing Secret, wrong mount path, or read-only permission issues.

Updating Secrets

Kubernetes does not provide an in-place update command for Secret data; you must apply a new manifest or use kubectl edit. When you update a Secret, mounts as volumes are eventually updated (by kubelet sync), but environment variables are not changed until the Pod restarts. To update a Secret from a manifest after modifying the YAML file:

kubectl apply -f db-secret.yaml

To update a single key via CLI, you can patch with JSON merge or strategic merge:

kubectl patch secret db-credentials -p '{"data":{"password":"bmV3cGFzcw=="}}' --namespace demo

Replace bmV3cGFzcw== with the base64 of your new password. Verify the update:

kubectl get secret db-credentials -o jsonpath='{.data.password}' --namespace demo | base64 --decode

Expected new password. If you need to rotate credentials, update the Secret, then restart the Pods that consume it:

kubectl rollout restart deployment.apps/myapp --namespace demo

For StatefulSets, a rolling update may be done by deleting Pods one by one.

Troubleshooting Secrets

Problems with Secrets often surface as application errors: authentication failures, missing files, or crash loops. Start with the Pod status:

kubectl get pods --namespace demo

If a Pod is CrashLoopBackOff, inspect logs:

kubectl logs secret-env-pod --namespace demo --previous

If the log indicates missing environment variable, check the Pod spec and the Secret existence:

kubectl get secret db-credentials -o yaml --namespace demo
kubectl describe pod secret-env-pod --namespace demo

Ensure the Secret is in the same namespace as the Pod; Secrets are namespaced objects. Also check RBAC: the ServiceAccount used by the Pod must have permission to get the Secret if the Secret is not allowed by the namespace default. By default, a Pod can mount any Secret in its namespace if the ServiceAccount has the get permission, which is often granted by the view role.

Failure Modes and Recovery

Common failure modes include:

1. Secret not found: Pod event shows MountVolume.SetUp failed for volume "credentials" : secret "db-credentials" not found. Confirm the Secret exists and is spelled correctly. If it was deleted accidentally, recreate it from a known good source.
2. Invalid base64 data: If the Secret's data cannot be decoded by Kubernetes (e.g., not base64), you will see an error when applying: Invalid value: "...": must be base64. Re-encode the value with base64 and re-apply.
3. Permission denied: Pod cannot read the Secret file because of runAsUser, fsGroup, or readOnly settings. Check the Pod security context and volume mount options.
4. Stale environment variables: After updating a Secret, environment variables remain old until Pod restart. Use kubectl rollout restart or delete the Pod to force recreation.
5. Secret too large: Kubernetes Secret has a 1 MiB limit. If you exceed it, the Secret creation fails. Use a smaller secret or split into multiple Secrets; for large blobs, consider a ConfigMap or external storage.

For recovery, always keep a backup of Secret manifests in a secure location (encrypted at rest). If a Secret is corrupted, delete and recreate from backup:

kubectl delete secret db-credentials --namespace demo
kubectl apply -f db-secret-backup.yaml

Then restart consuming Pods.

Common Pitfalls

• Using plain text in manifests: Never store plain text values in stringData for long-term; use data with base64 (though base64 is encoding, not encryption). For real security, integrate with a KMS provider or use Sealed Secrets.
• Wide access via default ServiceAccount: Avoid mounting the default ServiceAccount token automatically; set automountServiceAccountToken: false on the Pod spec if the app does not need API access.
• Forgetting to rotate Secrets: Secrets are static; without a rotation process, credentials become stale. Set an expiration reminder or use an external secret manager with automatic rotation.
• Not cleaning up test Secrets: Unused Secrets clutter the cluster and may expose sensitive data. Regularly list and delete unused ones:

kubectl get secrets --namespace demo
kubectl delete secret unwanted-secret --namespace demo

• Mixing namespaces: A Secret referenced by a Pod in a different namespace will fail because Secrets are namespace-scoped. If you need cross-namespace access, consider a shared external secret store or copy the Secret to the target namespace.

Operations Checklist

Use this checklist before and after any Secret operation:

Assign one accountable person for each step, and review the checklist monthly in a team meeting. The frequency column reflects how often the step should be validated.

Conclusion

Kubernetes Secrets require more than a few commands; they demand a disciplined workflow that observes, verifies, and recovers. By following the examples in this article, you can create, mount, update, and troubleshoot Secrets with confidence. Always remember to limit access, use least privilege, and treat Secrets as sensitive data even though they are base64 encoded. Your next step should be to practice these commands in a non-production namespace and then integrate Secret management into your CI/CD pipeline with encryption and rotation policies.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.