## Intro Operating ConfigMaps in production requires a disciplined, checklist-driven approach. A ConfigMap is a core Kubernetes object that decouples configuration from application code, allowing you to inject environment variables, command-line arguments, or configuration files into Pods without rebuilding container images. However, this flexibility introduces operational risks: a malformed ConfigMap can break multiple Deployments simultaneously, a missing key can cause crash loops, and unauthorized modification can expose sensitive settings. This article provides a practical operations checklist for Kubernetes ConfigMap production use, aimed at developers, DevOps consultants, and technical startup teams. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery paths. We will walk through concrete scenarios, commands, expected outputs, failure signals, and recovery decisions. Each section builds on the previous one, from environment inventory to safe configuration paths, verification, diagnostics, failure modes, and a consolidated checklist. Before diving in, ensure you have a running Kubernetes cluster (v1.21 or later is assumed) and kubectl configured with appropriate permissions. All commands are illustrative; replace resource names and namespaces with your own. ## Version and Environment Inventory Before making any change, establish the current state of your cluster, the ConfigMap resources, and the workloads consuming them. This inventory reduces the chance of affecting an unintended resource and provides a baseline for rollback. ### Verify Cluster and kubectl Version Check your cluster version and kubectl compatibility: ```bash kubectl version --client ``` Expected output includes Client Version. For example: ``` Client Version: v1.24.2 ``` If you need the server version, run `kubectl version` without the deprecated `--short` flag or use `kubectl get nodes`; however, for ConfigMaps, the v1 API has been stable since Kubernetes 1.6, so version skew is rarely an issue, but it is worth confirming client compatibility. ### Identify Namespaces and ConfigMaps List all ConfigMaps in the target namespace. Suppose your application runs in the `payment-processing` namespace: ```bash kubectl get configmaps -n payment-processing ``` Example output: ``` NAME DATA AGE app-config 4 12d db-config 2 30d logging-config 1 7d ``` Record the name, number of data keys, and age. An unexpectedly old ConfigMap may indicate a stale configuration that is still in use. ### Determine Which Workloads Use a ConfigMap Find Deployments, StatefulSets, DaemonSets, or Pods that reference a specific ConfigMap. For a ConfigMap named `app-config`: ```bash kubectl get deployments,statefulsets,daemonsets -n payment-processing -o json | jq '.items[] | select(.spec.template.spec.volumes[]?.configMap.name == "app-config" or .. | .configMapKeyRef?.name? == "app-config" or .configMapRef?.name? == "app-config") | {kind: .kind, name: .metadata.name}' ``` If you don't have jq, a simpler but less precise approach is to grep the YAML exports: ```bash kubectl get deployments -n payment-processing -o yaml | grep -B5 -A10 "app-config" ``` This helps you understand the blast radius of a ConfigMap change. For example, if three Deployments use `app-config`, a bad value could take down all three. ### Capture Current ConfigMap Content Export the ConfigMap to a backup file before any change: ```bash kubectl get configmap app-config -n payment-processing -o yaml > app-config-backup-$(date +%Y%m%d%H%M%S).yaml ``` This backup is essential for rollback. Store it in version control or a secure location. ### Prerequisites Check Ensure you have the following: - Permissions to get, list, and update ConfigMaps in the target namespace (RBAC roles `get`, `list`, `update`, `patch` on `configmaps`). - A clear understanding of which application version expects which configuration keys. If an app requires a new key not present in the ConfigMap, it may fail. - A maintenance window if the change is risky. ## Safe Configuration Path A safe configuration path means making changes in a way that minimizes risk. This involves using immutable practices where possible, testing in a staging namespace, and applying changes incrementally. Because ConfigMaps are not immutable by default, you must impose your own safety controls. ### Treat ConfigMaps as Immutable When Possible Since Kubernetes 1.21, you can set the `immutable` field on a ConfigMap to true. This prevents accidental updates and forces you to create a new ConfigMap and roll out a new Deployment when configuration changes. For critical production configs, this is a strong safeguard. Example immutable ConfigMap declaration: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: payment-gateway-config namespace: payment-processing immutable: true data: gateway.endpoint: "https://api.example.com" retry.maxAttempts: "3" ``` When you try to update an immutable ConfigMap, kubectl returns an error: ``` configmap/payment-gateway-config is immutable ``` You must then create a new ConfigMap (e.g., `payment-gateway-config-v2`) and update the Deployment to reference it. Rolling restart of the Pods is required to pick up the new config. This process makes the change explicit and reviewable. ### Use a Staging Namespace for Testing Before applying changes to production, test them in a namespace like `staging` or a dedicated development namespace. Create the same ConfigMap there and deploy a test Pod or Deployment that consumes it. Example: create a test ConfigMap from a file using a kustomization. Create a `kustomization.yaml` in a directory: ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization configMapGenerator: - name: app-config-test files: - config.properties ``` Then apply it to the staging namespace: ```bash kubectl apply -k . -n staging ``` This generates a ConfigMap with a hashed name based on the content. Use `kubectl get configmaps -n staging` to see the generated name. Then run a temporary Pod that prints the value of a key to verify: ```bash kubectl run config-test-pod --image=busybox -n staging --restart=Never --rm -it -- /bin/sh -c 'echo $CONFIG_KEY' ``` Set environment variables from the ConfigMap using `--env-from` or define them in a test Deployment. ### Apply Changes with kubectl apply and Record Intent Use `kubectl apply` instead of `kubectl edit` for production changes. Applying a YAML file from version control provides an audit trail. Before applying, review the diff: ```bash kubectl diff -f updated-app-config.yaml -n payment-processing ``` Expected output shows the differences, for example: ```diff --- /tmp/LIVE-123/configmap/app-config +++ /tmp/MERGED-456/configmap/app-config @@ -3,7 +3,7 @@ data: db.host: db.internal - db.port: "5432" + db.port: "5433" ``` If the diff looks correct, apply: ```bash kubectl apply -f updated-app-config.yaml -n payment-processing ``` Record the change in your change management system with a rollback plan. ### Limit Blast Radius by Using Separate ConfigMaps Do not put all configuration for different services into one giant ConfigMap. Instead, create one ConfigMap per application or per concern (e.g., `db-config`, `cache-config`, `app-config`). This way, changing the database connection string does not affect an unrelated service. ### Avoid Storing Secrets in ConfigMaps ConfigMaps are not encrypted at rest by default. Never put passwords, API keys, or tokens in ConfigMaps. Use Kubernetes Secrets for sensitive data, and consider external secret management tools like Vault or Sealed Secrets. If you find a secret in a ConfigMap, migrate it immediately and rotate the credential. Example of an unsafe ConfigMap entry: ```yaml data: database.password: "S3cr3t!" ``` Instead, store the password in a Secret and reference it in your Pod spec: ```yaml env: - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password ``` ## Verification and Diagnostics After applying a ConfigMap change, verify that it took effect correctly and that applications are healthy. Verification is an active process: do not assume success without checking. ### Verify ConfigMap Data Retrieve the ConfigMap and confirm the expected keys and values: ```bash kubectl get configmap app-config -n payment-processing -o yaml ``` Look for the updated data. If you used `kubectl apply`, the resource version will change. You can also use `kubectl describe` for a summary: ```bash kubectl describe configmap app-config -n payment-processing ``` ### Trigger a Rolling Update If your Deployment consumes the ConfigMap via environment variables or volume mounts, Pods will not automatically restart when the ConfigMap changes (unless you are using a tool like Reloader). To apply the new configuration, you must restart the Pods. For a Deployment named `payment-service`: ```bash kubectl rollout restart deployment/payment-service -n payment-processing ``` Monitor the rollout status: ```bash kubectl rollout status deployment/payment-service -n payment-processing ``` Expected output on success: ``` deployment "payment-service" successfully rolled out ``` If the rollout fails, the command exits non-zero and prints an error, such as: ``` error: deployment "payment-service" exceeded its progress deadline ``` ### Check Pod Logs and Environment Inspect Pods to ensure they started with the new configuration. List Pods: ```bash kubectl get pods -n payment-processing -l app=payment-service -o wide ``` Check logs for errors indicating configuration issues: ```bash kubectl logs deployment/payment-service -n payment-processing --tail=50 ``` If a specific Pod is crash looping, use `--previous`: ```bash kubectl logs --previous -n payment-processing ``` Verify the actual environment variable in a running Pod (if the ConfigMap is used as env var): ```bash kubectl exec -n payment-processing -- printenv DB_HOST ``` For volume-mounted config files, exec into the Pod and view the file: ```bash kubectl exec -n payment-processing -- cat /etc/config/db.properties ``` ### Use Port Forwarding for Local Verification If your application exposes an HTTP endpoint that reflects its configuration (like a health or info endpoint), you can forward a local port to the Pod and query it: ```bash kubectl port-forward deployment/payment-service 8080:8080 -n payment-processing ``` Then in another terminal: ```bash curl http://localhost:8080/health ``` Check that the response indicates healthy and correct configuration. ## Failure Modes and Recovery Even with careful planning, ConfigMap changes can cause failures. This section lists common failure modes and concrete recovery steps. ### Failure: ConfigMap Key Missing If an application expects a key that is not present in the ConfigMap, it may fail at startup with an error like `Environment variable DB_HOST not set`. **Diagnosis:** ```bash kubectl get configmap app-config -n payment-processing -o jsonpath='{.data}' ``` Compare with expected keys. **Recovery:** 1. Restore the key by applying a corrected ConfigMap. 2. If you have a backup, use `kubectl apply -f backup.yaml`. 3. Restart the Deployment: `kubectl rollout restart deployment/payment-service`. ### Failure: ConfigMap Reference Error If a Pod spec references a ConfigMap that does not exist, the Pod will not start, and the event log shows `MountVolume.SetUp failed for volume ... configmap not found`. **Diagnosis:** ```bash kubectl describe pod -n payment-processing ``` Look for events: ``` Warning FailedMount 10s (x10 over 2m) kubelet MountVolume.SetUp failed for volume "config-volume" : configmap "missing-config" not found ``` **Recovery:** - Create the missing ConfigMap with the correct name and data. - Or fix the Pod spec to reference the correct ConfigMap (requires updating the Deployment). ### Failure: ConfigMap Too Large ConfigMaps are stored in etcd and have a maximum size of 1 MiB (1,048,576 bytes). If you try to create or update a ConfigMap larger than that, the API server rejects it: ``` configmap "large-config" is invalid: metadata.annotations: Too long: must have at most 262144 bytes ``` This is a rare but possible error. It also can occur if the total metadata size exceeds limits. **Recovery:** - Split the configuration into multiple ConfigMaps. - For large files, consider using a PersistentVolume or an init container to fetch configuration from a remote source. ### Failure: Application Unresponsive After ConfigMap Update The application might start but behave incorrectly due to a bad configuration value (e.g., wrong database port). **Diagnosis:** - Check application logs for errors. - Verify the actual value in the Pod: `kubectl exec -- printenv DB_PORT`. - Compare with expected value. **Recovery:** - Roll back the ConfigMap to the previous version using backup file: ```bash kubectl apply -f app-config-backup-YYYYMMDDHHMMSS.yaml ``` - Restart the Deployment: `kubectl rollout restart deployment/payment-service`. - Monitor logs and health endpoints. ### Failure: Pod Stuck in CrashLoopBackOff The Pod repeatedly crashes because of an invalid configuration. The ConfigMap itself may be valid, but the application interprets it incorrectly. **Diagnosis:** ```bash kubectl get pods -n payment-processing ``` Look for `CrashLoopBackOff` status: ``` NAME READY STATUS RESTARTS AGE payment-service-6d4b7c9f8-abcde 0/1 CrashLoopBackOff 5 4m ``` Inspect logs: ```bash kubectl logs payment-service-6d4b7c9f8-abcde --previous -n payment-processing ``` Common error: `Invalid value for DB_PORT: 'abc'`. **Recovery:** - Correct the ConfigMap value. - If the deployment cannot be fixed quickly, consider rolling back to a previous known-good ConfigMap and restarting. - Use `kubectl scale deployment payment-service --replicas=0` to stop crash loops while fixing (only if the service can be down briefly). ## Operations Checklist Use the following checklist as a quick reference for every ConfigMap change in production. Each item includes a command or action and the expected result. ### Pre-Change - [ ] Verify cluster and kubectl versions: `kubectl version --client` (expect Client Version output; use `kubectl version` without `--short` for server version). - [ ] List target ConfigMaps: `kubectl get configmaps -n ` (record names, data counts, ages). - [ ] Identify consumers: use `grep` or `jq` to find Deployments referencing the ConfigMap. Document the blast radius. - [ ] Export backup: `kubectl get configmap -n -o yaml > backup.yaml`. Store backup securely. - [ ] Check for sensitive data: visually inspect ConfigMap YAML for passwords or tokens. If found, migrate to Secrets and rotate. - [ ] Review change in staging: apply new ConfigMap in `staging` namespace, test with a temporary Pod. - [ ] Prepare rollback plan: know the backup file location and restart procedure. ### Change - [ ] Edit ConfigMap YAML in version control (never use `kubectl edit` directly). - [ ] Preview diff: `kubectl diff -f updated.yaml -n ` (ensure only intended changes). - [ ] Apply: `kubectl apply -f updated.yaml -n `. - [ ] Record change in change management system. ### Post-Change - [ ] Verify ConfigMap content: `kubectl get configmap -n -o yaml | grep `. - [ ] Restart consumers: `kubectl rollout restart deployment/ -n ` for each affected Deployment. - [ ] Monitor rollout: `kubectl rollout status deployment/ -n ` (expect success). - [ ] Check Pod logs: `kubectl logs deployment/ -n --tail=50` for configuration errors. - [ ] Validate runtime config: `kubectl exec -- printenv ` or `kubectl exec -- cat /path/to/config/file`. - [ ] Test health endpoint or application functionality. - [ ] If failure, execute rollback: apply backup ConfigMap, restart deployment, verify recovery. ### Ongoing Maintenance - [ ] Periodically review ConfigMaps for unused or stale keys. - [ ] Use immutable ConfigMaps for critical configurations. - [ ] Implement GitOps for ConfigMap management if possible. - [ ] Monitor for ConfigMap size approaching 1 MiB limit. - [ ] Audit access: ensure only authorized users can modify ConfigMaps (RBAC). ## Conclusion Kubernetes ConfigMap production operations demand the same rigor as any infrastructure change. The checklist presented here emphasizes observation before intervention, secure handling of sensitive data, controlled change application, thorough verification, and clear recovery procedures. A ConfigMap is a simple object, but its misuse can cause widespread outages. By following this checklist, you reduce the risk of unintended consequences, shorten incident response time, and build a culture of safe configuration management. Remember: the operational goal is not just to make changes but to make them visible, reversible, and verifiable. As a next step, select one low-risk ConfigMap improvement from your current environment. Apply the inventory and safe configuration path practices, and document the outcome. Then extend the same discipline to other Kubernetes configuration resources, such as Secrets and Deployments, to create a consistent operational framework. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.