## Intro

Kubernetes ConfigMaps let you separate configuration from application code. A local lab gives you a safe place to learn how ConfigMaps behave, test changes, and debug issues without touching a shared cluster. This guide walks through a complete local setup using Minikube or kind, then shows practical examples for mounting ConfigMaps as environment variables, files, and command-line arguments. You will also learn how to verify changes, observe pod behavior, and recover from common mistakes.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

## Prerequisites

Before starting, ensure you have:

- A running local Kubernetes cluster (Minikube, kind, or Docker Desktop with Kubernetes).
- `kubectl` installed and configured to point to your local cluster.
- Basic familiarity with `kubectl get`, `kubectl describe`, and applying YAML files.

Verify your cluster is ready:

```bash
kubectl cluster-info
kubectl get nodes
```

Expected output shows a node in `Ready` status:

```
NAME       STATUS   ROLES           AGE   VERSION
minikube   Ready    control-plane   10m   v1.28.0
```

## Version and Environment Inventory

Always know your environment before making changes. Check the Kubernetes server and client versions, because ConfigMap features like `immutable` require Kubernetes 1.19 or newer.

```bash
kubectl version --short
```

Example output:

```
Client Version: v1.28.0
Kustomize Version: v5.0.1
Server Version: v1.28.0
```

List the namespaces and confirm you are working in the intended one (default for this lab):

```bash
kubectl get namespaces
kubectl config get-contexts
```

The current context shows the cluster and namespace. If you are not in the right context, switch with `kubectl config use-context`.

For ConfigMaps, verify the API resource exists:

```bash
kubectl api-resources | grep configmap
```

Expected output:

```
configmaps                cm           v1                                     true         ConfigMap
```

Record these details before any change. They help when asking for help or reverting.

## Creating Your First ConfigMap

Create a simple ConfigMap from a YAML file. Save the following as `app-config.yaml`:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
  namespace: default
data:
  APP_COLOR: "blue"
  APP_MODE: "development"
  app.properties: |
    key1=value1
    key2=value2
  nginx.conf: |
    server {
      listen 80;
      server_name localhost;
      location / {
        root /usr/share/nginx/html;
      }
    }
```

Apply it:

```bash
kubectl apply -f app-config.yaml
```

Verify the ConfigMap was created:

```bash
kubectl get configmap app-config -o yaml
```

Or view a summarized version:

```bash
kubectl describe configmap app-config
```

Expected output shows `Name: app-config`, `Namespace: default`, and the data entries.

You can also create a ConfigMap from command line literals:

```bash
kubectl create configmap game-config --from-literal=GAME_MODE=classic --from-literal=PLAYER_COUNT=4
```

Verify with:

```bash
kubectl get configmap game-config -o yaml
```

## Using ConfigMaps in a Pod

Now mount the ConfigMap into a pod as environment variables and as files. Save the following as `pod-using-configmap.yaml`:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: configmap-demo-pod
spec:
  containers:
  - name: demo-container
    image: busybox:1.36
    command: ["/bin/sh", "-c", "echo APP_COLOR=$APP_COLOR; echo APP_MODE=$APP_MODE; cat /etc/config/app.properties; sleep 3600"]
    env:
    - name: APP_COLOR
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: APP_COLOR
    - name: APP_MODE
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: APP_MODE
    volumeMounts:
    - name: config-volume
      mountPath: /etc/config
  volumes:
  - name: config-volume
    configMap:
      name: app-config
```

Apply and wait for the pod to be running:

```bash
kubectl apply -f pod-using-configmap.yaml
kubectl wait --for=condition=Ready pod/configmap-demo-pod --timeout=60s
```

Check the pod logs to see the values:

```bash
kubectl logs configmap-demo-pod
```

Expected output:

```
APP_COLOR=blue
APP_MODE=development
key1=value1
key2=value2
```

To inspect the mounted files inside the container:

```bash
kubectl exec configmap-demo-pod -- ls /etc/config
kubectl exec configmap-demo-pod -- cat /etc/config/app.properties
```

## Passing ConfigMap Values as Command-Line Arguments

You can use ConfigMap keys in container command arguments. Modify the pod spec to include arguments referencing the ConfigMap. Save as `pod-with-args.yaml`:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: configmap-arg-pod
spec:
  containers:
  - name: arg-container
    image: busybox:1.36
    command: ["/bin/sh", "-c"]
    args:
    - "echo Starting in $GAME_MODE mode with $PLAYER_COUNT players; sleep 3600"
    env:
    - name: GAME_MODE
      valueFrom:
        configMapKeyRef:
          name: game-config
          key: GAME_MODE
    - name: PLAYER_COUNT
      valueFrom:
        configMapKeyRef:
          name: game-config
          key: PLAYER_COUNT
```

Apply and check logs:

```bash
kubectl apply -f pod-with-args.yaml
kubectl logs configmap-arg-pod
```

Expected output:

```
Starting in classic mode with 4 players
```

## Updating ConfigMaps and Propagation Delay

ConfigMaps are updated independently of pods. When you update a ConfigMap, pods using it as environment variables do not automatically see the new value; pods using it as a mounted volume may see changes after a delay (usually up to a minute, depending on the kubelet sync period).

Update `app-config` to change the color:

```bash
kubectl patch configmap app-config --type merge -p '{"data":{"APP_COLOR":"green"}}'
```

Check the running pod's environment variable (it remains blue):

```bash
kubectl exec configmap-demo-pod -- sh -c 'echo $APP_COLOR'
```

Output: `blue`

Check the mounted file (may still show old or new value depending on timing):

```bash
kubectl exec configmap-demo-pod -- cat /etc/config/app.properties
```

To force a pod to pick up updated environment variables, you must restart the pod:

```bash
kubectl delete pod configmap-demo-pod
kubectl apply -f pod-using-configmap.yaml
kubectl wait --for=condition=Ready pod/configmap-demo-pod --timeout=60s
kubectl logs configmap-demo-pod
```

Now the output shows `APP_COLOR=green`.

## Using Immutable ConfigMaps

If your configuration never changes, mark the ConfigMap as immutable to protect it and improve performance. To create an immutable ConfigMap:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: immutable-config
immutable: true
data:
  STATIC_SETTING: "never-changes"
```

Apply and try to patch it:

```bash
kubectl apply -f immutable-config.yaml
kubectl patch configmap immutable-config --type merge -p '{"data":{"STATIC_SETTING":"new-value"}}'
```

Expected error:

```
configmap "immutable-config" is immutable
```

This is useful for preventing accidental edits.

## Verifying ConfigMap Mounts and Environment Variables

Always verify that your pod sees the expected configuration. Use `kubectl exec` to inspect from inside the container.

Check environment variables:

```bash
kubectl exec configmap-demo-pod -- env | grep APP_
```

Check mounted file contents and permissions:

```bash
kubectl exec configmap-demo-pod -- ls -l /etc/config
```

Expected output shows the files with default permissions (usually 0644). You can change permissions using `defaultMode` in the volume spec:

```yaml
volumes:
- name: config-volume
  configMap:
    name: app-config
    defaultMode: 0440
```

Also verify that only expected keys are present. If you mount the entire ConfigMap as a volume, all keys become files. To mount only specific keys, use `items`:

```yaml
volumes:
- name: config-volume
  configMap:
    name: app-config
    items:
    - key: app.properties
      path: my-app.properties
```

Now `/etc/config` contains only `my-app.properties`.

## Debugging Common Issues

### Pod Stuck in CrashLoopBackOff

If your pod crashes, inspect the logs and events:

```bash
kubectl logs configmap-demo-pod --previous
kubectl describe pod configmap-demo-pod
```

Common causes:

- ConfigMap key referenced in `env` does not exist. The pod creation fails with `Invalid value: "...": key not found in ConfigMap`.
- Mounted file path is incorrect or conflicts with an existing file.
- Command syntax error due to missing environment variable.

Fix by correcting the ConfigMap or pod spec, then delete the pod and reapply.

### ConfigMap Not Found

If `kubectl apply` returns an error that the ConfigMap does not exist, ensure you created it in the same namespace as the pod. ConfigMaps are namespace-scoped. Check with `kubectl get configmap -n <namespace>`.

### Pod Cannot Access Mounted Files

Check the volumeMount path and ensure the container does not run as a non-root user without permissions. Use `securityContext` to adjust if needed.

## Failure Modes and Recovery

ConfigMaps are simple but can cause subtle issues if not managed carefully.

**Failure 1: Missing key reference.** If a pod references a ConfigMap key that does not exist, the pod will fail to start with a `CreateContainerConfigError` event. Recover by adding the key to the ConfigMap or fixing the reference, then delete the pod to restart.

**Failure 2: Large ConfigMap.** ConfigMaps are limited to 1 MiB (as of Kubernetes 1.28). Exceeding this limit causes an error when creating the ConfigMap. Split large configs into multiple ConfigMaps or use a volume from a Secret or external storage.

**Failure 3: Accidental deletion.** If a ConfigMap is deleted while pods use it, existing pods continue running, but new pods or restarted containers fail to mount the ConfigMap. Recover by restoring the ConfigMap from your version control or backup, then delete the affected pods to force recreation.

**Failure 4: Immutable ConfigMap modification.** As shown earlier, immutable ConfigMaps cannot be changed. If you need to modify, create a new ConfigMap and update pod references.

## Operations Checklist

Use this checklist before and after making ConfigMap changes in a production-like environment:

- [ ] Confirm cluster version and namespace: `kubectl version --short && kubectl config view --minify`
- [ ] List existing ConfigMaps: `kubectl get configmaps -n <namespace>`
- [ ] Back up current ConfigMap: `kubectl get configmap <name> -o yaml > configmap-backup.yaml`
- [ ] Apply changes: `kubectl apply -f <file>`
- [ ] Verify ConfigMap content: `kubectl describe configmap <name>`
- [ ] Restart dependent pods: `kubectl rollout restart deployment/<deployment-name>` if using Deployments
- [ ] Check pod logs for startup errors: `kubectl logs <pod-name>`
- [ ] Confirm environment variables: `kubectl exec <pod-name> -- env | grep <key>`
- [ ] Confirm mounted files: `kubectl exec <pod-name> -- cat /path/to/file`
- [ ] Monitor for up to 5 minutes to ensure no delayed issues

## Conclusion

A Kubernetes ConfigMap local lab gives you the confidence to handle configuration changes safely. By following the examples and verification steps in this guide, you can isolate issues, understand propagation behavior, and recover quickly from failures. Practice with different volume mounts, environment variable references, and immutable ConfigMaps to deepen your operational knowledge.

As a next step, choose one low-risk verification for your own application, record the current state, apply a ConfigMap change, and observe how your pods behave. Document your own checklist for future incidents. With ConfigMaps managed correctly, your deployments become more maintainable and less error-prone.