Kubernetes Kustomization Configuration Mistakes and How to Avoid Them

Introduction

Kustomize is a Kubernetes-native configuration management tool that lets you customize raw YAML files without templates. It works by layering patches, resources, and generators on top of a base configuration. When used correctly, Kustomize reduces duplication and keeps environments consistent. When misconfigured, it can cause failed deployments, unexpected resource changes, and hard-to-debug issues in production.

This article walks through the most common Kustomization configuration mistakes, explains why they happen, and shows how to avoid or recover from them. Each mistake includes a concrete example, the commands to diagnose it, and the steps to fix it. The focus is on practical, operational safety: observe before changing, limit blast radius, verify results, and document recovery paths.

Before diving into mistakes, let's establish a baseline environment and the key commands used throughout.

Version and Environment Baseline

Before debugging any Kustomize issue, confirm your tool versions and cluster context. Kustomize is built into kubectl as kubectl kustomize, but a standalone kustomize binary is also available. Features and behavior can vary between versions.

Check your kubectl version and the embedded Kustomize version:

kubectl version --client
# Output includes:
# Client Version: v1.28.0
# Kustomize Version: v5.0.3

If you use the standalone binary:

kustomize version
# Output:
# v5.0.3

Verify your current Kubernetes context and namespace:

kubectl config current-context
# Example: gke_my-project_us-central1-a_cluster-1
kubectl config view --minify | grep namespace

A common mistake is applying Kustomize output to the wrong cluster or namespace because the context was not checked. Always confirm context before any apply.

For all examples, we'll use a simple base directory structure:

base/
  kustomization.yaml
  deployment.yaml
  service.yaml
overlays/
  dev/
    kustomization.yaml
    patch.yaml
  prod/
    kustomization.yaml
    patch.yaml

Note: The filename kustomization.yaml is the default, but kustomization.yml and Kustomization are also recognized. Using an unsupported filename is a frequent mistake.

Safe Configuration Path: Building Incrementally

A safe workflow for Kustomize is to build and inspect the rendered manifests before applying them to the cluster. This allows you to catch mistakes early without affecting running workloads.

Step-by-step safe path

1. Write your base and overlay files.
2. Run kubectl kustomize <directory> to render the final manifests.
3. Review the output for expected changes.
4. Optionally, pipe to kubectl diff to see what would change in the cluster.
5. Apply using kubectl apply -k <directory>.
6. Verify with kubectl rollout status and other checks.

Example of rendering an overlay:

kubectl kustomize overlays/dev
# Outputs the full YAML with all patches applied.

Example of diffing against the live cluster:

kubectl diff -k overlays/dev
# Shows lines added/removed compared to current cluster state.

This approach prevents many mistakes because you see exactly what will be applied.

Common Kustomization Mistakes and How to Avoid Them

Mistake 1: Incorrect file paths or missing resources

Kustomize references files in the resources field, patches field, and other generators. A typo in a file path leads to a build error or, worse, silently ignoring a patch.

Example:

In overlays/dev/kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
patchesStrategicMerge:
  - deployment-patch.yaml  # File does not exist, actual name is patch.yaml

When you run kubectl kustomize overlays/dev, you'll see an error like:

Error: accumulating resources: accumulation err='accumulating resources from 'deployment-patch.yaml': evalsymlink failure on '/path/overlays/dev/deployment-patch.yaml' : lstat /path/overlays/dev/deployment-patch.yaml: no such file or directory'

How to avoid:

• Double-check file paths relative to the kustomization file.
• Use kubectl kustomize or kustomize build after any file rename.
• Consider using directory structure that keeps overlays close to their patches.

Recovery:

Fix the path and rebuild. The error is caught before applying, so no cluster impact.

Mistake 2: Misunderstanding patch semantics (strategic merge vs JSON patches)

Kustomize supports several patch types: patchesStrategicMerge, patchesJson6902, and the newer patches field with patch and target. Each has different behavior. Using the wrong type can result in unexpected resource mutations.

Example: Strategic merge patch that unintentionally deletes a field

Base deployment has:

spec:
  template:
    spec:
      containers:
        - name: app
          image: nginx:1.19
          ports:
            - containerPort: 80

Overlay patch patch.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      containers:
        - name: app
          image: nginx:1.21

The patch omits ports. With strategic merge, the ports list from the base is retained because it's a list of maps with a merge key (containerPort). However, if you used a JSON 6902 patch with "op": "replace" on the container, it might remove fields not specified. Understanding the merge semantics is critical.

How to avoid:

• Prefer patches with explicit target and patch for clarity.
• Test patch results with kubectl kustomize and review the full output.
• For complex modifications, consider using patchesJson6902 only when you need precise JSON Pointer operations.

Recovery:

If a patch removed something unintentionally, fix the patch and reapply. If already applied, you may need to reapply the base configuration or manually restore fields.

Mistake 3: Improper use of generators (ConfigMaps, Secrets)

Kustomize generators create resources from files or literals. A common mistake is using configMapGenerator without a generatorOptions to set a hash suffix, leading to immutable ConfigMap issues when updating data.

Example:

kustomization.yaml:

configMapGenerator:
  - name: app-config
    files:
      - config.properties

By default, Kustomize appends a content hash to the ConfigMap name, e.g., app-config-7b58b6d6f5. When the file content changes, the name changes, forcing a rolling update of Deployments that reference it. This is often desired. However, if you set generatorOptions.disableNameSuffixHash: true, the name stays constant, and updates to the ConfigMap won't trigger pod restarts unless you handle it manually.

Mistake: Disabling hash suffix without understanding the consequence. Pods may keep using old config until restarted.

How to avoid:

• Leave hash suffix enabled unless you have a deliberate reason.
• If you disable it, implement a strategy to restart workloads (e.g., kubectl rollout restart deployment/app).
• Use kubectl get configmap to see generated names.

Recovery:

If pods are not picking up new config, manually restart the deployment:

kubectl rollout restart deployment/app

Mistake 4: Overriding namespaces incorrectly

Kustomize can set the namespace for all resources using the namespace field. But a common mistake is assuming it applies to cluster-scoped resources or to resources that already have a namespace hard-coded.

Example:

namespace: my-namespace
resources:
  - deployment.yaml
  - service.yaml

This sets metadata.namespace on the deployment and service, but not on a ClusterRole or Namespace resource if included. Also, if a base resource already has metadata.namespace: default, the overlay's namespace does not override it; instead, you need a patch.

How to avoid:

• Use namespace at overlay level consistently.
• Check rendered output to see namespaces.
• Use kubectl get all -n my-namespace to verify.

Recovery:

If resources ended up in the wrong namespace, you can move them by extracting and reapplying with correct namespace, or delete and recreate.

Mistake 5: Misusing replacements and variable substitution

Kustomize has replacements and vars (deprecated) to substitute values across resources. vars are deprecated and often lead to confusion because they only work in certain fields. Many users still use vars and encounter errors or unexpected results.

Example using deprecated vars:

vars:
  - name: SERVICE_NAME
    objref:
      kind: Service
      name: my-service
      apiVersion: v1
    fieldref:
      fieldpath: metadata.name

Then in a ConfigMap, you might try $(SERVICE_NAME) but it won't be substituted unless the field supports it. replacements are more powerful and safer.

How to avoid:

• Use replacements instead of vars.
• Understand which fields support replacements.
• Always build and inspect output for substitutions.

Recovery:

If substitution didn't happen, check the replacements configuration. Fix and rebuild.

Verification and Diagnostics

After applying Kustomize output, verify that the cluster state matches expectations. Here are essential diagnostic commands.

Check rollout status

kubectl rollout status deployment/app -n my-namespace
# Output:
# Waiting for deployment "app" rollout to finish: 1 old replicas are pending termination...
# deployment "app" successfully rolled out

Examine pods

kubectl get pods -n my-namespace -o wide
# Check READY, STATUS, RESTARTS, AGE, IP, NODE

For a failing pod:

kubectl describe pod <pod-name> -n my-namespace
# Look at Events section for image pull errors, scheduling failures, etc.

Check logs:

kubectl logs <pod-name> -n my-namespace
# For previous crash:
kubectl logs <pod-name> -n my-namespace --previous

Compare rendered vs applied

If you suspect a mismatch, render the kustomization again and diff with what's in the cluster:

kubectl kustomize overlays/dev > /tmp/rendered.yaml
kubectl get deployment app -n my-namespace -o yaml > /tmp/live.yaml
diff /tmp/rendered.yaml /tmp/live.yaml

This helps spot unintentional changes.

Failure Modes and Recovery

Even with careful configuration, failures happen. Here are common failure modes specific to Kustomize and how to recover.

Failure: Apply fails due to invalid manifest

Symptom: kubectl apply -k overlays/dev returns an error like:

error: error validating "overlays/dev": error validating data: [ValidationError(Deployment.spec): unknown field "replicas" in io.k8s.api.apps.v1.DeploymentSpec, ...]

Cause: The rendered manifest has an invalid field, perhaps due to a patch adding a field that's not supported by the API version.

Recovery:

• Run kubectl kustomize overlays/dev and inspect the output.
• Look for the invalid field and correct the patch or base.
• Reapply.

Failure: Rollout does not complete

Symptom: Pods are created but deployment never reaches desired replica count. kubectl rollout status hangs.

Cause: Possibly the new image is wrong, resource limits are too low, or readiness probe fails.

Recovery:

• Check pod events: kubectl describe pod <new-pod>.
• Check logs for application errors.
• If needed, rollback to previous version using kubectl rollout undo deployment/app.
• Or apply a previous kustomization overlay.

Failure: ConfigMap or Secret not updated

Symptom: After updating a ConfigMap generator file, pods still have old configuration.

Cause: Hash suffix disabled, or deployment not restarted.

Recovery:

• Check ConfigMap name: kubectl get configmap -n my-namespace.
• If name unchanged, manually restart: kubectl rollout restart deployment/app.
• Consider enabling hash suffix for future updates.

Operations Checklist for Kustomize Deployments

Use this checklist before and after applying Kustomize changes to production. The checklist owner (e.g., the release engineer) should review each item and sign off.

Pre-apply checklist

• [ ] Cluster context and namespace confirmed (kubectl config current-context).
• [ ] Rendered manifests reviewed (kubectl kustomize overlays/prod).
• [ ] kubectl diff -k overlays/prod shows expected changes only.
• [ ] Secrets are not committed to Git; placeholders or external secret management used.
• [ ] Rollback plan documented (e.g., previous Git commit hash, kubectl rollout undo).
• [ ] Resource quotas and limits are sufficient.
• [ ] Backup of any critical resources exists.

Post-apply checklist

• [ ] kubectl apply -k overlays/prod succeeded without errors.
• [ ] Rollout status complete: kubectl rollout status deployment/app -n prod.
• [ ] Pods healthy: kubectl get pods -n prod shows all Running.
• [ ] Logs show no fatal errors (spot check).
• [ ] Service endpoints responding: kubectl get endpoints -n prod and test a request.
• [ ] Metrics and alerts are nominal.
• [ ] Document any deviations for next review.

Owner: Priya Sharma, Release Engineering Lead.

Review frequency: Every production release; post-incident reviews additionally.

Conclusion

Kustomize is a powerful tool, but its flexibility means configuration mistakes are easy to make. By understanding common pitfalls—such as incorrect file paths, patch type confusion, generator misconfigurations, namespace overrides, and deprecated variable substitution—you can avoid most issues before they reach production.

The key practices are: inspect rendered output with kubectl kustomize, diff against the cluster, apply incrementally, and verify with rollout status and logs. Always have a rollback plan and document the owner and review cadence.

Start by applying these practices in a development environment. Create a small overlay, introduce a deliberate mistake, and observe the error. Then fix it and build confidence. Over time, these habits will make your Kustomize deployments safer and more predictable.

Next step: Take one of your existing kustomization directories and run kubectl kustomize on it. Review the output carefully. Then run kubectl diff -k to see what would change in the cluster. This simple exercise can reveal hidden issues in your current configuration.