## Introduction

Kustomize is a Kubernetes-native configuration management tool that lets you customize raw YAML files without templates. It works by layering patches, resources, and generators on top of a base configuration. When used correctly, Kustomize reduces duplication and keeps environments consistent. When misconfigured, it can cause failed deployments, unexpected resource changes, and hard-to-debug issues in production.

This article walks through the most common Kustomization configuration mistakes, explains why they happen, and shows how to avoid or recover from them. Each mistake includes a concrete example, the commands to diagnose it, and the steps to fix it. The focus is on practical, operational safety: observe before changing, limit blast radius, verify results, and document recovery paths.

Before diving into mistakes, let's establish a baseline environment and the key commands used throughout.

## Version and Environment Baseline

Before debugging any Kustomize issue, confirm your tool versions and cluster context. Kustomize is built into `kubectl` as `kubectl kustomize`, but a standalone `kustomize` binary is also available. Features and behavior can vary between versions.

Check your `kubectl` version and the embedded Kustomize version:

```bash
kubectl version --client
# Output includes:
# Client Version: v1.28.0
# Kustomize Version: v5.0.3
```

If you use the standalone binary:

```bash
kustomize version
# Output:
# v5.0.3
```

Verify your current Kubernetes context and namespace:

```bash
kubectl config current-context
# Example: gke_my-project_us-central1-a_cluster-1
kubectl config view --minify | grep namespace
```

A common mistake is applying Kustomize output to the wrong cluster or namespace because the context was not checked. Always confirm context before any apply.

For all examples, we'll use a simple base directory structure:

```
base/
  kustomization.yaml
  deployment.yaml
  service.yaml
overlays/
  dev/
    kustomization.yaml
    patch.yaml
  prod/
    kustomization.yaml
    patch.yaml
```

Note: The filename `kustomization.yaml` is the default, but `kustomization.yml` and `Kustomization` are also recognized. Using an unsupported filename is a frequent mistake.

## Safe Configuration Path: Building Incrementally

A safe workflow for Kustomize is to build and inspect the rendered manifests before applying them to the cluster. This allows you to catch mistakes early without affecting running workloads.

### Step-by-step safe path

1. Write your base and overlay files.
2. Run `kubectl kustomize <directory>` to render the final manifests.
3. Review the output for expected changes.
4. Optionally, pipe to `kubectl diff` to see what would change in the cluster.
5. Apply using `kubectl apply -k <directory>`.
6. Verify with `kubectl rollout status` and other checks.

Example of rendering an overlay:

```bash
kubectl kustomize overlays/dev
# Outputs the full YAML with all patches applied.
```

Example of diffing against the live cluster:

```bash
kubectl diff -k overlays/dev
# Shows lines added/removed compared to current cluster state.
```

This approach prevents many mistakes because you see exactly what will be applied.

## Common Kustomization Mistakes and How to Avoid Them

### Mistake 1: Incorrect file paths or missing resources

Kustomize references files in the `resources` field, `patches` field, and other generators. A typo in a file path leads to a build error or, worse, silently ignoring a patch.

**Example:**

In `overlays/dev/kustomization.yaml`:

```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patchesStrategicMerge:
  - deployment-patch.yaml  # File does not exist, actual name is patch.yaml
```

When you run `kubectl kustomize overlays/dev`, you'll see an error like:

```
Error: accumulating resources: accumulation err='accumulating resources from 'deployment-patch.yaml': evalsymlink failure on '/path/overlays/dev/deployment-patch.yaml' : lstat /path/overlays/dev/deployment-patch.yaml: no such file or directory'
```

**How to avoid:**

- Double-check file paths relative to the kustomization file.
- Use `kubectl kustomize` or `kustomize build` after any file rename.
- Consider using directory structure that keeps overlays close to their patches.

**Recovery:**

Fix the path and rebuild. The error is caught before applying, so no cluster impact.

### Mistake 2: Misunderstanding patch semantics (strategic merge vs JSON patches)

Kustomize supports several patch types: `patchesStrategicMerge`, `patchesJson6902`, and the newer `patches` field with `patch` and `target`. Each has different behavior. Using the wrong type can result in unexpected resource mutations.

**Example: Strategic merge patch that unintentionally deletes a field**

Base deployment has:

```yaml
spec:
  template:
    spec:
      containers:
        - name: app
          image: nginx:1.19
          ports:
            - containerPort: 80
```

Overlay patch `patch.yaml`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      containers:
        - name: app
          image: nginx:1.21
```

The patch omits `ports`. With strategic merge, the `ports` list from the base is retained because it's a list of maps with a merge key (containerPort). However, if you used a JSON 6902 patch with `"op": "replace"` on the container, it might remove fields not specified. Understanding the merge semantics is critical.

**How to avoid:**

- Prefer `patches` with explicit `target` and `patch` for clarity.
- Test patch results with `kubectl kustomize` and review the full output.
- For complex modifications, consider using `patchesJson6902` only when you need precise JSON Pointer operations.

**Recovery:**

If a patch removed something unintentionally, fix the patch and reapply. If already applied, you may need to reapply the base configuration or manually restore fields.

### Mistake 3: Improper use of generators (ConfigMaps, Secrets)

Kustomize generators create resources from files or literals. A common mistake is using `configMapGenerator` without a `generatorOptions` to set a hash suffix, leading to immutable ConfigMap issues when updating data.

**Example:**

`kustomization.yaml`:

```yaml
configMapGenerator:
  - name: app-config
    files:
      - config.properties
```

By default, Kustomize appends a content hash to the ConfigMap name, e.g., `app-config-7b58b6d6f5`. When the file content changes, the name changes, forcing a rolling update of Deployments that reference it. This is often desired. However, if you set `generatorOptions.disableNameSuffixHash: true`, the name stays constant, and updates to the ConfigMap won't trigger pod restarts unless you handle it manually.

**Mistake:** Disabling hash suffix without understanding the consequence. Pods may keep using old config until restarted.

**How to avoid:**

- Leave hash suffix enabled unless you have a deliberate reason.
- If you disable it, implement a strategy to restart workloads (e.g., `kubectl rollout restart deployment/app`).
- Use `kubectl get configmap` to see generated names.

**Recovery:**

If pods are not picking up new config, manually restart the deployment:

```bash
kubectl rollout restart deployment/app
```

### Mistake 4: Overriding namespaces incorrectly

Kustomize can set the namespace for all resources using the `namespace` field. But a common mistake is assuming it applies to cluster-scoped resources or to resources that already have a namespace hard-coded.

**Example:**

```yaml
namespace: my-namespace
resources:
  - deployment.yaml
  - service.yaml
```

This sets `metadata.namespace` on the deployment and service, but not on a ClusterRole or Namespace resource if included. Also, if a base resource already has `metadata.namespace: default`, the overlay's namespace does not override it; instead, you need a patch.

**How to avoid:**

- Use `namespace` at overlay level consistently.
- Check rendered output to see namespaces.
- Use `kubectl get all -n my-namespace` to verify.

**Recovery:**

If resources ended up in the wrong namespace, you can move them by extracting and reapplying with correct namespace, or delete and recreate.

### Mistake 5: Misusing replacements and variable substitution

Kustomize has `replacements` and `vars` (deprecated) to substitute values across resources. `vars` are deprecated and often lead to confusion because they only work in certain fields. Many users still use `vars` and encounter errors or unexpected results.

**Example using deprecated vars:**

```yaml
vars:
  - name: SERVICE_NAME
    objref:
      kind: Service
      name: my-service
      apiVersion: v1
    fieldref:
      fieldpath: metadata.name
```

Then in a ConfigMap, you might try `$(SERVICE_NAME)` but it won't be substituted unless the field supports it. `replacements` are more powerful and safer.

**How to avoid:**

- Use `replacements` instead of `vars`.
- Understand which fields support replacements.
- Always build and inspect output for substitutions.

**Recovery:**

If substitution didn't happen, check the `replacements` configuration. Fix and rebuild.

## Verification and Diagnostics

After applying Kustomize output, verify that the cluster state matches expectations. Here are essential diagnostic commands.

### Check rollout status

```bash
kubectl rollout status deployment/app -n my-namespace
# Output:
# Waiting for deployment "app" rollout to finish: 1 old replicas are pending termination...
# deployment "app" successfully rolled out
```

### Examine pods

```bash
kubectl get pods -n my-namespace -o wide
# Check READY, STATUS, RESTARTS, AGE, IP, NODE
```

For a failing pod:

```bash
kubectl describe pod <pod-name> -n my-namespace
# Look at Events section for image pull errors, scheduling failures, etc.
```

Check logs:

```bash
kubectl logs <pod-name> -n my-namespace
# For previous crash:
kubectl logs <pod-name> -n my-namespace --previous
```

### Compare rendered vs applied

If you suspect a mismatch, render the kustomization again and diff with what's in the cluster:

```bash
kubectl kustomize overlays/dev > /tmp/rendered.yaml
kubectl get deployment app -n my-namespace -o yaml > /tmp/live.yaml
diff /tmp/rendered.yaml /tmp/live.yaml
```

This helps spot unintentional changes.

## Failure Modes and Recovery

Even with careful configuration, failures happen. Here are common failure modes specific to Kustomize and how to recover.

### Failure: Apply fails due to invalid manifest

**Symptom:** `kubectl apply -k overlays/dev` returns an error like:

```
error: error validating "overlays/dev": error validating data: [ValidationError(Deployment.spec): unknown field "replicas" in io.k8s.api.apps.v1.DeploymentSpec, ...]
```

**Cause:** The rendered manifest has an invalid field, perhaps due to a patch adding a field that's not supported by the API version.

**Recovery:**

- Run `kubectl kustomize overlays/dev` and inspect the output.
- Look for the invalid field and correct the patch or base.
- Reapply.

### Failure: Rollout does not complete

**Symptom:** Pods are created but deployment never reaches desired replica count. `kubectl rollout status` hangs.

**Cause:** Possibly the new image is wrong, resource limits are too low, or readiness probe fails.

**Recovery:**

- Check pod events: `kubectl describe pod <new-pod>`.
- Check logs for application errors.
- If needed, rollback to previous version using `kubectl rollout undo deployment/app`.
- Or apply a previous kustomization overlay.

### Failure: ConfigMap or Secret not updated

**Symptom:** After updating a ConfigMap generator file, pods still have old configuration.

**Cause:** Hash suffix disabled, or deployment not restarted.

**Recovery:**

- Check ConfigMap name: `kubectl get configmap -n my-namespace`.
- If name unchanged, manually restart: `kubectl rollout restart deployment/app`.
- Consider enabling hash suffix for future updates.

## Operations Checklist for Kustomize Deployments

Use this checklist before and after applying Kustomize changes to production. The checklist owner (e.g., the release engineer) should review each item and sign off.

### Pre-apply checklist

- [ ] Cluster context and namespace confirmed (`kubectl config current-context`).
- [ ] Rendered manifests reviewed (`kubectl kustomize overlays/prod`).
- [ ] `kubectl diff -k overlays/prod` shows expected changes only.
- [ ] Secrets are not committed to Git; placeholders or external secret management used.
- [ ] Rollback plan documented (e.g., previous Git commit hash, `kubectl rollout undo`).
- [ ] Resource quotas and limits are sufficient.
- [ ] Backup of any critical resources exists.

### Post-apply checklist

- [ ] `kubectl apply -k overlays/prod` succeeded without errors.
- [ ] Rollout status complete: `kubectl rollout status deployment/app -n prod`.
- [ ] Pods healthy: `kubectl get pods -n prod` shows all Running.
- [ ] Logs show no fatal errors (spot check).
- [ ] Service endpoints responding: `kubectl get endpoints -n prod` and test a request.
- [ ] Metrics and alerts are nominal.
- [ ] Document any deviations for next review.

**Owner:** Priya Sharma, Release Engineering Lead.

**Review frequency:** Every production release; post-incident reviews additionally.

## Conclusion

Kustomize is a powerful tool, but its flexibility means configuration mistakes are easy to make. By understanding common pitfalls—such as incorrect file paths, patch type confusion, generator misconfigurations, namespace overrides, and deprecated variable substitution—you can avoid most issues before they reach production.

The key practices are: inspect rendered output with `kubectl kustomize`, diff against the cluster, apply incrementally, and verify with rollout status and logs. Always have a rollback plan and document the owner and review cadence.

Start by applying these practices in a development environment. Create a small overlay, introduce a deliberate mistake, and observe the error. Then fix it and build confidence. Over time, these habits will make your Kustomize deployments safer and more predictable.

Next step: Take one of your existing kustomization directories and run `kubectl kustomize` on it. Review the output carefully. Then run `kubectl diff -k` to see what would change in the cluster. This simple exercise can reveal hidden issues in your current configuration.