## Intro

Secrets in Kubernetes are objects that store sensitive information such as passwords, OAuth tokens, and SSH keys. Using Secrets instead of plain text in Pod specs or container images reduces the risk of accidental exposure and makes credential rotation more manageable. However, many engineers only memorize a few kubectl commands and do not practice the observation, verification, and recovery steps that turn those commands into reliable operations.

This article covers the core kubectl commands for creating, inspecting, mounting, and troubleshooting Secrets. It is written for developers, DevOps consultants, and startup teams who need practical examples with clear expected output and failure signals. Each section includes commands, sample outputs, and what to check when something does not behave as expected. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of real secrets, and verify every change.

You need a working Kubernetes cluster and kubectl configured to follow along. The examples assume a namespace named `demo`; create it with `kubectl create namespace demo` if it does not exist. Some commands use `base64` for encoding, available on macOS, Linux, and Windows Subsystem for Linux.

## Version and Environment Inventory

Before running any Secret commands, confirm your cluster version and API availability. Kubernetes versions 1.19 and later include the stable `v1` Secret API, but older clusters may require `v1beta1`. The `kubectl version` command shows both client and server versions:

```bash
kubectl version --short
```

Sample output:

```text
Client Version: v1.28.2
Server Version: v1.27.3
```

If the server version is older than 1.19, upgrade or use the appropriate API version in your manifests. Check that you can read Secrets in the target namespace with a non-destructive command:

```bash
kubectl auth can-i list secrets --namespace demo
```

Expected output when permission exists:

```text
yes
```

If it returns `no`, ask your cluster administrator to grant read access via a Role or ClusterRole. Also verify the default ServiceAccount token is not being used for secret access if you have disabled auto-mounting:

```bash
kubectl get serviceaccount default -o yaml --namespace demo | grep automountServiceAccountToken
```

If nothing is returned, auto-mounting is enabled by default and may be a security consideration. A read-only environment inventory ensures that you do not attempt changes your role cannot support.

## Creating Secrets

Kubernetes supports several ways to create Secrets: from literal values, from files, and from a YAML manifest. The `kubectl create secret generic` command is quick for simple cases, but it writes the secret value into your shell history. Use it only for development; in production, prefer declarative manifests applied via CI/CD where the manifest source is protected.

### From Literal Values

Create a Secret with username and password literals:

```bash
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password='S3cret!' \
  --namespace demo
```

Expected output:

```text
secret/db-credentials created
```

Verify the Secret was created:

```bash
kubectl get secret db-credentials --namespace demo
```

Output:

```text
NAME             TYPE     DATA   AGE
db-credentials   Opaque   2      10s
```

### From Files

If you already have credential files, create a Secret from them:

```bash
echo -n 'admin' > username.txt
echo -n 'S3cret!' > password.txt
kubectl create secret generic db-credentials-file \
  --from-file=./username.txt \
  --from-file=./password.txt \
  --namespace demo
```

The keys will be the filenames (`username.txt`, `password.txt`). To specify custom keys, use `--from-file=username=./username.txt --from-file=password=./password.txt`.

Inspect the resulting data keys:

```bash
kubectl get secret db-credentials-file -o jsonpath='{.data}' --namespace demo
```

Output (base64 encoded):

```text
{"password.txt":"UzNjcmV0IQ==","username.txt":"YWRtaW4="}
```

### From a Manifest

For production, write a YAML file with base64-encoded values:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: db-credentials-manifest
  namespace: demo
type: Opaque
data:
  username: YWRtaW4=  # admin
  password: UzNjcmV0IQ==  # S3cret!
```

Apply it:

```bash
kubectl apply -f db-secret.yaml
```

Expected output:

```text
secret/db-credentials-manifest created
```

If the Secret already exists, `apply` updates it (subject to validation). You can generate the base64 values with:

```bash
echo -n 'admin' | base64
# YWRtaW4=
echo -n 'S3cret!' | base64
# UzNjcmV0IQ==
```

Never commit raw manifest files with real base64 values to a public repository; use sealed secrets or an external secret manager for encryption.

## Reading and Decoding Secrets

To see what is inside a Secret without decoding manually, use `kubectl get secret -o yaml` and then decode the base64 values:

```bash
kubectl get secret db-credentials-manifest -o yaml --namespace demo
```

Output snippet:

```yaml
data:
  password: UzNjcmV0IQ==
  username: YWRtaW4=
kind: Secret
metadata:
  ...
```

Decode a value:

```bash
echo 'UzNjcmV0IQ==' | base64 --decode
# S3cret!
```

For quick extraction of a single value:

```bash
kubectl get secret db-credentials-manifest \
  -o jsonpath='{.data.password}' --namespace demo | base64 --decode
```

*Caution*: Decoding secrets on your workstation exposes them in terminal output. Limit this to troubleshooting and avoid piping to logs or shared screens.

## Mounting Secrets as Environment Variables

A Pod can consume Secrets as environment variables. This is common for application configuration. Create a Pod manifest:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: secret-env-pod
  namespace: demo
spec:
  containers:
  - name: app
    image: busybox:1.28
    command: ["sleep", "3600"]
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASS
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password
```

Apply and verify:

```bash
kubectl apply -f secret-env-pod.yaml
kubectl exec secret-env-pod -- namespace demo -- printenv DB_USER
```

Expected output:

```text
admin
```

If the environment variable is missing, check the Pod events for invalid Secret references:

```bash
kubectl describe pod secret-env-pod --namespace demo
```

Look for events like `Error: secret "db-credentials" not found`.

## Mounting Secrets as Volumes

Files provide a dynamic way to consume Secrets; changes to the Secret are reflected in the mounted volume (though the application must reload the file). Example Pod with a volume mount:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: secret-volume-pod
  namespace: demo
spec:
  containers:
  - name: app
    image: busybox:1.28
    command: ["sleep", "3600"]
    volumeMounts:
    - name: credentials
      mountPath: /etc/credentials
      readOnly: true
  volumes:
  - name: credentials
    secret:
      secretName: db-credentials
```

After applying, check the files:

```bash
kubectl exec secret-volume-pod --namespace demo -- ls /etc/credentials
# password  username
kubectl exec secret-volume-pod --namespace demo -- cat /etc/credentials/username
# admin
```

If the mount fails, the Pod will not start. Check the Pod status and events:

```bash
kubectl get pod secret-volume-pod --namespace demo
kubectl describe pod secret-volume-pod --namespace demo
```

Common errors include missing Secret, wrong mount path, or read-only permission issues.

## Updating Secrets

Kubernetes does not provide an in-place update command for Secret data; you must apply a new manifest or use `kubectl edit`. When you update a Secret, mounts as volumes are eventually updated (by kubelet sync), but environment variables are not changed until the Pod restarts. To update a Secret from a manifest after modifying the YAML file:

```bash
kubectl apply -f db-secret.yaml
```

To update a single key via CLI, you can patch with JSON merge or strategic merge:

```bash
kubectl patch secret db-credentials -p '{"data":{"password":"bmV3cGFzcw=="}}' --namespace demo
```

Replace `bmV3cGFzcw==` with the base64 of your new password. Verify the update:

```bash
kubectl get secret db-credentials -o jsonpath='{.data.password}' --namespace demo | base64 --decode
```

Expected new password. If you need to rotate credentials, update the Secret, then restart the Pods that consume it:

```bash
kubectl rollout restart deployment.apps/myapp --namespace demo
```

For StatefulSets, a rolling update may be done by deleting Pods one by one.

## Troubleshooting Secrets

Problems with Secrets often surface as application errors: authentication failures, missing files, or crash loops. Start with the Pod status:

```bash
kubectl get pods --namespace demo
```

If a Pod is `CrashLoopBackOff`, inspect logs:

```bash
kubectl logs secret-env-pod --namespace demo --previous
```

If the log indicates missing environment variable, check the Pod spec and the Secret existence:

```bash
kubectl get secret db-credentials -o yaml --namespace demo
kubectl describe pod secret-env-pod --namespace demo
```

Ensure the Secret is in the same namespace as the Pod; Secrets are namespaced objects. Also check RBAC: the ServiceAccount used by the Pod must have permission to get the Secret if the Secret is not allowed by the namespace default. By default, a Pod can mount any Secret in its namespace if the ServiceAccount has the `get` permission, which is often granted by the `view` role.

## Failure Modes and Recovery

Common failure modes include:

1. **Secret not found**: Pod event shows `MountVolume.SetUp failed for volume "credentials" : secret "db-credentials" not found`. Confirm the Secret exists and is spelled correctly. If it was deleted accidentally, recreate it from a known good source.
2. **Invalid base64 data**: If the Secret's data cannot be decoded by Kubernetes (e.g., not base64), you will see an error when applying: `Invalid value: "...": must be base64`. Re-encode the value with `base64` and re-apply.
3. **Permission denied**: Pod cannot read the Secret file because of runAsUser, fsGroup, or readOnly settings. Check the Pod security context and volume mount options.
4. **Stale environment variables**: After updating a Secret, environment variables remain old until Pod restart. Use `kubectl rollout restart` or delete the Pod to force recreation.
5. **Secret too large**: Kubernetes Secret has a 1 MiB limit. If you exceed it, the Secret creation fails. Use a smaller secret or split into multiple Secrets; for large blobs, consider a ConfigMap or external storage.

For recovery, always keep a backup of Secret manifests in a secure location (encrypted at rest). If a Secret is corrupted, delete and recreate from backup:

```bash
kubectl delete secret db-credentials --namespace demo
kubectl apply -f db-secret-backup.yaml
```

Then restart consuming Pods.

## Common Pitfalls

- **Using plain text in manifests**: Never store plain text values in `stringData` for long-term; use `data` with base64 (though base64 is encoding, not encryption). For real security, integrate with a KMS provider or use Sealed Secrets.
- **Wide access via default ServiceAccount**: Avoid mounting the default ServiceAccount token automatically; set `automountServiceAccountToken: false` on the Pod spec if the app does not need API access.
- **Forgetting to rotate Secrets**: Secrets are static; without a rotation process, credentials become stale. Set an expiration reminder or use an external secret manager with automatic rotation.
- **Not cleaning up test Secrets**: Unused Secrets clutter the cluster and may expose sensitive data. Regularly list and delete unused ones:

```bash
kubectl get secrets --namespace demo
kubectl delete secret unwanted-secret --namespace demo
```

- **Mixing namespaces**: A Secret referenced by a Pod in a different namespace will fail because Secrets are namespace-scoped. If you need cross-namespace access, consider a shared external secret store or copy the Secret to the target namespace.

## Operations Checklist

Use this checklist before and after any Secret operation:

| Step | Command / Action | Expected Result | Owner | Review Frequency |
|------|------------------|-----------------|-------|------------------|
| 1. Verify cluster access | `kubectl auth can-i create secrets --namespace demo` | `yes` | Priya Shah, Engineering Lead | Monthly |
| 2. Inspect existing Secrets | `kubectl get secrets --namespace demo` | List of Secrets with age | Alex Chen, DevOps Engineer | Weekly |
| 3. Backup current Secret | `kubectl get secret <name> -o yaml > secret-backup.yaml` | Backup file created | Alex Chen, DevOps Engineer | Daily |
| 4. Create or update Secret | `kubectl apply -f secret.yaml` | `secret/<name> created` or `configured` | Priya Shah, Engineering Lead | Per release |
| 5. Verify Secret data | `kubectl get secret <name> -o jsonpath='{.data}'` | Base64 values present | Sam Patel, App Developer | Per change |
| 6. Restart consuming Pods | `kubectl rollout restart deployment <deployment>` | Pod restart initiated | Sam Patel, App Developer | After Secret update |
| 7. Monitor application logs | `kubectl logs <pod> --namespace demo` | No authentication errors | Sam Patel, App Developer | Post-deployment |
| 8. Remove temporary test Secrets | `kubectl delete secret test-secret` | Secret deleted | Alex Chen, DevOps Engineer | Monthly cleanup |

Assign one accountable person for each step, and review the checklist monthly in a team meeting. The frequency column reflects how often the step should be validated.

## Conclusion

Kubernetes Secrets require more than a few commands; they demand a disciplined workflow that observes, verifies, and recovers. By following the examples in this article, you can create, mount, update, and troubleshoot Secrets with confidence. Always remember to limit access, use least privilege, and treat Secrets as sensitive data even though they are base64 encoded. Your next step should be to practice these commands in a non-production namespace and then integrate Secret management into your CI/CD pipeline with encryption and rotation policies.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.