Helm Architecture Explained with Practical Examples

Intro

Helm is the package manager for Kubernetes, but to operate it safely you need to understand its architecture, not just memorize commands. This article explains Helm architecture with practical examples, connecting components, data flow, design, and operations to concrete commands, expected output, failure signals, and recovery decisions. It is written for developers, DevOps consultants, and technical startup teams who need to diagnose and fix Helm issues in real clusters.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before touching anything, know exactly what you are running. Helm's behavior changes between major versions (v2 vs v3 is a common source of confusion), and Kubernetes cluster versions add another layer. Start with these read-only commands:

# Helm client version
helm version --short
# Expected output example: v3.12.0+g0a3158d

# Kubernetes cluster version (requires kubectl)
kubectl version --short
# Expected output example: Client Version: v1.27.1 / Server Version: v1.26.3

# List current Helm releases in the default namespace
helm list --namespace default
# Expected output example:
# NAME  	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART       	APP VERSION
# myapp 	default  	1        	2023-06-01 10:00:00.000000 +0000 UTC	deployed	myapp-0.1.0	1.16.0

These commands are safe: they do not modify cluster state. Record the output in your runbook or terminal log before any intervention.

Identify the deployment topology. Are you using Tiller (Helm v2) or the pure client/server model (Helm v3)? Helm v3 removed Tiller; if you see a tiller pod in kube-system, you are on v2 and should plan a migration. Check with:

kubectl get pods -n kube-system | grep tiller
# No output means no Tiller (Helm v3 or later), or Tiller not installed.

Prerequisites for this document: Helm v3.8+ installed, kubectl v1.24+ with a valid kubeconfig, and a test namespace where you can create resources. Use explicit placeholders in examples: <release-name>, <chart-name>, <namespace>. Never place real credentials or production identifiers in an article or in your terminal history.

Helm Architecture: Core Components

Helm v3 consists of two main parts: the Helm client and the Kubernetes API server. Unlike v2, there is no in-cluster server component (Tiller) storing release information. Instead, release state is stored as Kubernetes Secrets (by default) or ConfigMaps in the cluster itself.

Helm client (helm): The command-line tool you run locally or in CI/CD. It renders templates, interacts with the Kubernetes API, and manages release history.

Chart: A package of pre-configured Kubernetes resources. Charts are directories or tarballs containing Chart.yaml (metadata), values.yaml (default configuration), and template files under templates/.

Repository: An HTTP server hosting packaged charts (index.yaml + .tgz files). Common public repositories include Bitnami, Prometheus Community, and Artifact Hub (a meta-repository).

Release: A specific instance of a chart running in a Kubernetes cluster. Each release has a name, a revision number, and stored configuration and manifest data.

Kubernetes Secrets/ConfigMaps as storage: When you install a release, Helm stores its metadata and rendered manifest in a Secret (default) or ConfigMap in the namespace of the release. You can inspect it:

# List secrets created by Helm for a release (name prefix is the release name)
kubectl get secrets -n <namespace> -l owner=helm
# Example output:
# NAME                          TYPE                DATA   AGE
# myapp.v1                     helm.sh/release.v1   1      3m

# Decode the release info (Helm v3 encodes it as a gzipped JSON in 'release' key)
kubectl get secret myapp.v1 -n <namespace> -o jsonpath='{.data.release}' | base64 -d | gzip -d | jq .

Understanding this storage is crucial for backup, migration, and debugging.

Helm Data Flow: From Chart to Running Resources

When you execute helm install myapp ./mychart -n mynamespace, the following steps occur:

1. Chart loading: Helm reads the chart directory or downloads from a repository. It validates Chart.yaml and loads default values from values.yaml.
2. Value merging: User-supplied values (-f file or --set) are merged over chart defaults. The merged values are accessible in templates via the .Values object.
3. Template rendering: Helm processes all files under templates/ using Go templates and the Sprig function library. It resolves {{ .Values.key }} and functions like {{ include }}, {{ toYaml }}, etc. Invalid templates cause an immediate error before contacting the cluster.
4. Resource validation: Helm parses the rendered YAML into Kubernetes objects and performs basic schema validation.
5. Resource ordering: Helm sorts resources by kind (e.g., Namespace, ServiceAccount, Secret, ConfigMap, Deployment, Service) to apply them in a sensible order. Custom ordering can be set with helm.sh/hook annotations.
6. API calls: Helm sends POST or PUT requests to the Kubernetes API server to create or update each resource.
7. Release storage: On success, Helm stores the release metadata and manifest as a Secret in the release namespace. If install fails, Helm may roll back (if --atomic is used) or leave partial resources (with --wait false).

Example of a simple deployment template:

# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-nginx
  labels:
    app: {{ .Release.Name }}-nginx
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Release.Name }}-nginx
  template:
    metadata:
      labels:
        app: {{ .Release.Name }}-nginx
    spec:
      containers:
        - name: nginx
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          ports:
            - containerPort: 80

With values.yaml:

replicaCount: 2
image:
  repository: nginx
  tag: "1.23"

Render the template to preview:

helm template myapp ./mychart --namespace mynamespace
# Output shows the complete Kubernetes YAML with all values substituted.

This dry-run does not contact the cluster and is invaluable for debugging templates.

Helm Design: Hooks, Lifecycle, and Release Management

Helm's design centers on release lifecycle management. Key concepts:

• Hooks: Allow intervention at certain points of a release lifecycle. For example, pre-install, post-install, pre-upgrade, post-upgrade, pre-rollback, post-rollback, pre-delete, post-delete. Hooks are Kubernetes resources annotated with helm.sh/hook: <hook-name>. They run to completion before the main resources are applied, or after.
• Release revisions: Every helm install, helm upgrade, or helm rollback increments the revision number. You can view history:

helm history myapp -n mynamespace
# Expected output:
# REVISION	UPDATED                 	STATUS    	CHART           	APP VERSION	DESCRIPTION
# 1       	Mon Jun  1 10:00:00 2023	deployed  	myapp-0.1.0     	1.16.0      	Install complete

• Rollback: Helm can roll back to a previous revision:

helm rollback myapp 1 -n mynamespace
# Expected output: Rollback was a success! Happy Helming!

Rollback is not a guarantee if the environment changed; verify the resulting state with kubectl get.

• Atomic and wait flags: --atomic rolls back on failure; --wait waits for pods to be ready. Use --atomic --wait --timeout 5m for production installs to avoid partial states.

Example hook job:

# templates/db-migration-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ .Release.Name }}-db-migrate
  annotations:
    "helm.sh/hook": pre-upgrade
    "helm.sh/hook-weight": "5"
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migration
          image: "{{ .Values.migration.image }}"
          command: ["./migrate.sh"]

Safe Configuration Path

Never store secrets in plain values files committed to Git. Helm supports external secret management via plugins like helm-secrets (sops), or by integrating with Kubernetes Secrets and referencing them from templates.

A safe configuration path involves:

1. Use values.yaml for non-sensitive defaults, but override with environment-specific files.
2. Use --set for simple overrides, but avoid complex --set chains; use --set-file or -f with structured YAML for readability.
3. For sensitive data, use a secret store and inject as environment variables or mounted files.

Example with a placeholder:

# values.yaml
postgres:
  host: mypostgres.default.svc.cluster.local
  port: 5432
  username: app_user
  # password should not be here; use existing secret
  passwordSecretName: myapp-postgres-secret
  passwordSecretKey: password

Template:

# templates/deployment.yaml
events:
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: {{ .Values.postgres.passwordSecretName }}
        key: {{ .Values.postgres.passwordSecretKey }}

Create the secret separately:

kubectl create secret generic myapp-postgres-secret \
  --from-literal=password='S3cureP@ssw0rd' -n mynamespace

Then install:

helm install myapp ./mychart -f values.yaml -n mynamespace

Verification: check pod environment variable not logged, and that secret exists:

kubectl get secret myapp-postgres-secret -n mynamespace -o yaml
# Output shows base64-encoded password; decode only if needed and via secure method.

Blast radius: if secret is compromised, rotate via kubectl create secret with new value and restart pods (kubectl rollout restart deployment myapp).

Verification and Diagnostics

After any Helm operation, verify the actual cluster state matches intent. Use read-only commands first.

Check release status:

helm status myapp -n mynamespace
# Output includes last deployed time, resources, and notes.

Check pod status:

kubectl get pods -n mynamespace -l app=myapp-nginx
# Expected: all pods Running and Ready

Check events:

kubectl get events -n mynamespace --sort-by=.metadata.creationTimestamp
# Look for warnings: FailedScheduling, ImagePullBackOff, CrashLoopBackOff, etc.

Check logs:

kubectl logs -n mynamespace deployment/myapp-nginx --tail=50

Debugging templating issues:

Use helm lint to check chart for issues:

helm lint ./mychart
# Example output:
# ==> Linting ./mychart
# [INFO] Chart.yaml: icon is recommended
# 1 chart(s) linted, 0 chart(s) failed

Use helm template --debug to see rendered templates and values:

helm template myapp ./mychart -n mynamespace --debug

Check release metadata:

helm get values myapp -n mynamespace
# Shows user-supplied values
helm get manifest myapp -n mynamespace
# Shows the manifest that was applied

Failure Modes and Recovery

Common Helm failures and how to recover:

1. Image Pull Errors

Symptom: Pod stuck in ImagePullBackOff or ErrImagePull.

Diagnosis:

kubectl describe pod myapp-nginx-<hash> -n mynamespace
# Look at Events: Failed to pull image "nginx:1.23": rpc error: code = NotFound desc = failed to pull and unpack image...

Cause: Incorrect image tag, repository credentials missing, or registry unreachable.

Recovery: Correct values.yaml image tag or create image pull secret. Then upgrade:

helm upgrade myapp ./mychart -f values.yaml -n mynamespace

Verify pod pulls and runs.

2. Failed Upgrades with Partial Resources

Symptom: helm upgrade fails, leaving some new pods and old pods running, or a mixture.

Cause: Invalid new template, missing required value, or resource conflict.

Recovery: Use --atomic to automatically rollback on failure:

helm upgrade myapp ./mychart -f new-values.yaml -n mynamespace --atomic --wait --timeout 5m

If already in bad state, rollback manually:

helm rollback myapp <previous-revision> -n mynamespace

Then fix the chart and try again.

3. Hook Failures

Symptom: Upgrade hangs, and helm status shows a hook job failed.

Diagnosis:

kubectl get jobs -n mynamespace
# Look for hook job in failed state.
kubectl describe job myapp-db-migrate -n mynamespace

Recovery: Fix the hook script or permissions, then delete the failed job and re-run upgrade:

kubectl delete job myapp-db-migrate -n mynamespace
helm upgrade myapp ./mychart -f values.yaml -n mynamespace

4. Namespace Not Found

Symptom: helm install fails with Error: namespaces "mynewnamespace" not found.

Cause: Helm does not create namespaces by default (unless --create-namespace is used).

Recovery: Either create namespace first:

kubectl create namespace mynewnamespace
helm install myapp ./mychart -n mynewnamespace

Or use --create-namespace flag:

helm install myapp ./mychart -n mynewnamespace --create-namespace

5. Release Metadata Corruption

Symptom: helm list shows release in unknown state, or commands fail with Error: release: not found.

Cause: Release secret deleted, or manual changes messed up storage.

Recovery: If secret exists but corrupted, you may need to reconstruct from history or use helm upgrade --force to recreate resources. If secret missing, reinstall as new release if possible. Prevention: always backup release secrets or use helm get values and helm get manifest before risky operations.

Operations Checklist

Use this checklist before any Helm operation in production.

1. Capture current state (read-only):

   helm list -n <namespace>
   helm history <release> -n <namespace>
   helm get values <release> -n <namespace> > values-backup.yaml
   helm get manifest <release> -n <namespace> > manifest-backup.yaml

1. Verify chart and values with helm lint and helm template --debug.

1. Check cluster connectivity and RBAC:

   kubectl auth can-i create deployments -n <namespace>
   # Expected: yes

1. Dry-run upgrade (if applicable):

   helm upgrade --dry-run --debug <release> ./chart -f new-values.yaml -n <namespace>

1. Apply change with atomic and wait for production:

   helm upgrade <release> ./chart -f new-values.yaml -n <namespace> --atomic --wait --timeout 5m

1. Verify deployment (pods, services, endpoints, ingress):

   kubectl get pods,svc,ingress -n <namespace>
   kubectl rollout status deployment/<deployment-name> -n <namespace>

1. Check logs and events for warnings.

1. Document the change in your change management system with revision number, timestamp, and operator.

1. Test recovery path in a non-production environment regularly.

Example workflow for upgrading a web app:

# Backup
helm get values webapp -n prod > webapp-values-backup-20230601.yaml
helm get manifest webapp -n prod > webapp-manifest-backup-20230601.yaml

# Dry-run
helm upgrade --dry-run --debug webapp ./webapp -f new-values.yaml -n prod

# Apply
helm upgrade webapp ./webapp -f new-values.yaml -n prod --atomic --wait --timeout 5m

# Verify
kubectl rollout status deployment/webapp -n prod
kubectl get pods -n prod -l app=webapp

Expected verification: deployment "webapp" successfully rolled out and all pods 1/1 Running.

Conclusion

Helm architecture explained with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

You now have a foundation: inventory environment, understand components and data flow, manage safe configuration, verify changes, diagnose failures, and follow an operations checklist. As a next step, choose one low-risk verification from this article, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Kubernetes version and your CI/CD setup.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.