Automating Helm with CI/CD gives you fast, repeatable, and reversible Kubernetes deployments. Done well, it also reduces rework by catching errors before they reach the cluster and by providing clean rollback paths when something slips through. This guide focuses on a safe, observable path you can implement today: - Establish clear version and environment inventory so every step runs against predictable targets. - Adopt a staged rollout approach with guarded Helm flags. - Use small, composable CI stages (lint, render, validate, deploy) and environment-specific values. - Verify success with concrete checks and artifacts. - Prepare for failure with tested rollback and recovery commands. By the end, you will have practical examples for GitHub Actions and GitLab CI/CD, a working set of commands, and a repeatable checklist you can adapt to your team's services. ## Version and Environment Inventory Before you write a single pipeline job, inventory your runtime. Consistent versions are critical for reproducible renders and reliable deployments. Example environment inventory:
ComponentVersionNotes
Helm CLIv3.14.xConsistent across local and CI runners
Kubernetesv1.27.xAPI stability assumed in manifests
kubectlv1.27.xMatch server minor version
Chart APIapiVersion: v2Required for Helm 3 packaging
Namespace layoutapp-staging, app-prodOne namespace per environment
Minimum prerequisites: - You have cluster access for staging and production (separate Kubernetes contexts or separate clusters). - You can create namespaces and RBAC for a CI service account with permissions limited to the target namespaces. - Your Helm chart(s) use Chart.yaml apiVersion: v2 and include a values.schema.json when feasible to validate inputs. Quick version checks: helm version kubectl version --short If versions diverge across dev machines and CI, pin them in your CI jobs and in developer onboarding docs. ## Safe Configuration Path The goal is predictable, low-risk promotion from a preflighted change to a healthy production release. Use the following implementation choices as a safe baseline. ### Namespaces and Release Names - Staging: namespace app-staging , release name myapp-staging - Production: namespace app-prod , release name myapp ### Values Files - Keep environment-specific values files: values/staging.yaml , values/production.yaml - Pin image tags (no latest ) and include replica counts per environment. ### Guarded Upgrades Use atomic upgrades, explicit timeouts, and history retention: helm upgrade --install myapp-staging ./charts/myapp \ --namespace app-staging \ --create-namespace \ --values values/staging.yaml \ --atomic \ --timeout 5m \ --history-max 10 --atomic rolls back on failure, preventing half-applied states. Keep timeouts tight in staging to surface slow rollouts early, and slightly higher in production to account for scale. ### Readiness and Health Gates - Ensure Deployments/StatefulSets have readinessProbes that reflect real availability. - Add Kubernetes annotations and labels your monitoring uses to track SLOs. ### Test Hooks Add a simple Helm test that verifies the app answers basic traffic. Example test in templates/tests/test-connection.yaml : apiVersion: v1 kind: Pod metadata: name: "{{ include \"myapp.fullname\" . }}-test-connection" annotations: "helm.sh/hook": test spec: restartPolicy: Never containers: - name: curl image: curlimages/curl:8.7.1 command: ["sh", "-c"] args: - | set -eu echo "Pinging service..." curl -fsS http://{{ include "myapp.fullname" . }}:{{ .Values.service.port }}/healthz ### Schema Validation Include values.schema.json to validate user-provided values. Lint and dry-run will fail early if a key is missing or of the wrong type. ## Practical Automation Examples The core pattern is the same across platforms: - Preflight: helm lint and helm template . - Staging deploy: helm upgrade --install with --atomic and a staging values file. - Health checks: kubectl rollout and helm test . - Promote to production with the same chart and pinned image tag. Below are two concise examples. Adapt versions and paths to your repo. ### Example 1: GitHub Actions This workflow runs on pushes to main. It lints, renders, does a dry-run, deploys to staging, verifies, and then permits a manual promotion to production. name: helm-cicd on: push: branches: [ main ] workflow_dispatch: {} jobs: preflight: runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v4 - name: Install Helm run: | curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash - name: Lint chart run: helm lint charts/myapp - name: Render manifests (template) run: helm template myapp ./charts/myapp -f values/staging.yaml > rendered.yaml - name: Dry-run upgrade run: helm upgrade --install myapp-staging ./charts/myapp -n app-staging -f values/staging.yaml --dry-run --debug deploy-staging: needs: preflight runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v4 - name: Install Helm run: | curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash - name: Configure kubeconfig run: | mkdir -p ~/.kube echo "$KUBECONFIG_CONTENT" > ~/.kube/config env: KUBECONFIG_CONTENT: ${{ secrets.STAGING_KUBECONFIG }} - name: Deploy to staging (atomic) run: | helm upgrade --install myapp-staging ./charts/myapp \ -n app-staging --create-namespace \ -f values/staging.yaml --atomic --timeout 5m --history-max 20 - name: Verify rollout run: | kubectl -n app-staging rollout status deploy/myapp --timeout=120s helm -n app-staging test myapp-staging --logs promote-production: if: ${{ github.event_name == 'workflow_dispatch' }} needs: deploy-staging runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v4 - name: Install Helm run: | curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash - name: Configure kubeconfig run: | mkdir -p ~/.kube echo "$KUBECONFIG_CONTENT" > ~/.kube/config env: KUBECONFIG_CONTENT: ${{ secrets.PROD_KUBECONFIG }} - name: Deploy to production (atomic) run: | helm upgrade --install myapp ./charts/myapp \ -n app-prod --create-namespace \ -f values/production.yaml --atomic --timeout 10m --history-max 50 - name: Verify rollout run: | kubectl -n app-prod rollout status deploy/myapp --timeout=300s helm -n app-prod test myapp --logs Notes: - Store kubeconfigs or OIDC-based auth securely (example uses encrypted secrets). - The production job is gated behind a manual workflow_dispatch . Replace with your preferred approval gate. ### Example 2: GitLab CI/CD A simple, stage-based pipeline: preflight -> staging -> production. stages: - preflight - staging - production variables: HELM_HISTORY_MAX: "30" .prep: &prep before_script: - apt-get update && apt-get install -y curl ca-certificates - curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash preflight: stage: preflight image: debian:12-slim <<: *prep script: - helm lint charts/myapp - helm template myapp ./charts/myapp -f values/staging.yaml > rendered.yaml - helm upgrade --install myapp-staging ./charts/myapp -n app-staging -f values/staging.yaml --dry-run --debug artifacts: paths: - rendered.yaml staging: stage: staging image: debian:12-slim <<: *prep script: - kubectl config set-cluster staging --server=$KUBE_SERVER --certificate-authority=/ca.crt - kubectl config set-credentials ci --token=$KUBE_TOKEN - kubectl config set-context staging --cluster=staging --user=ci --namespace=app-staging - kubectl config use-context staging - helm upgrade --install myapp-staging ./charts/myapp -n app-staging -f values/staging.yaml --atomic --timeout 5m --history-max $HELM_HISTORY_MAX - kubectl -n app-staging rollout status deploy/myapp --timeout=120s - helm -n app-staging test myapp-staging --logs dependencies: - preflight rules: - if: $CI_PIPELINE_SOURCE == "push" production: stage: production image: debian:12-slim when: manual allow_failure: false <<: *prep script: - kubectl config set-cluster prod --server=$KUBE_SERVER --certificate-authority=/ca.crt - kubectl config set-credentials ci --token=$KUBE_TOKEN - kubectl config set-context prod --cluster=prod --user=ci --namespace=app-prod - kubectl config use-context prod - helm upgrade --install myapp ./charts/myapp -n app-prod -f values/production.yaml --atomic --timeout 10m --history-max $HELM_HISTORY_MAX - kubectl -n app-prod rollout status deploy/myapp --timeout=300s - helm -n app-prod test myapp --logs dependencies: - staging Notes: - Inject Kubernetes credentials via masked CI variables or use an identity provider. - Artifacts ( rendered.yaml ) help during reviews and incident postmortems. ## Verification and Diagnostics Verification should be observable and fast. Add these steps to both developer playbooks and CI logs. ### Lint, Schema, and Render helm lint charts/myapp # catches many chart issues helm template myapp charts/myapp -f values/staging.yaml | tee rendered.yaml Expected: lint passes; rendered.yaml contains only supported API versions for your cluster. ### Dry-Run Upgrade with Debug Output helm upgrade --install myapp-staging charts/myapp \ -n app-staging -f values/staging.yaml --dry-run --debug Expected: exit code 0; manifests printed; hooks show planned execution. ### Apply and Watch Rollout helm upgrade --install myapp-staging charts/myapp \ -n app-staging -f values/staging.yaml --atomic --timeout 5m kubectl -n app-staging rollout status deploy/myapp --timeout=120s Expected: "Deployment successfully rolled out" message. ### Run Helm Tests helm -n app-staging test myapp-staging --logs Expected: test pod completes with phase Succeeded and returns expected output. ### Inspect Live State If Something Looks Off kubectl -n app-staging get pods,svc,ingress kubectl -n app-staging describe deploy/myapp kubectl -n app-staging get events --sort-by=.lastTimestamp | tail -n 50 kubectl -n app-staging logs deploy/myapp --all-containers --tail=200 ### Confirm What Helm Applied helm -n app-staging get manifest myapp-staging | less helm -n app-staging history myapp-staging These commands, combined with artifacts ( rendered.yaml ) and CI logs, form your primary triage toolkit. ## Failure Modes and Recovery Things will fail. The key is to fail fast in staging and recover quickly anywhere. Use the table below as a quick reference.
SymptomLikely CauseImmediate Action
Upgrade times outReadiness probe failing or pods unschedulablekubectl describe pods; review probes; increase timeout only after fix
ImagePullBackOffWrong tag or missing registry credentialsVerify image tag in values; check imagePullSecrets and registry access
Hook failedTest or pre-install hook errorhelm -n <ns> get hooks <release>; fix hook template or command, re-run
Stuck PendingInstallHook blocked or CRD issuekubectl get jobs/hooks; inspect logs; uninstall release and re-deploy
CRD upgrade failsIncompatible CRD changeApply CRD updates first; break change into two releases
### Rollback and Recovery Commands Fast rollback to last good revision: helm -n app-prod history myapp helm -n app-prod rollback myapp 12 --wait --atomic - Pick the last successful revision from history. - --atomic ensures rollback itself rolls back if it fails. Clean up a failed install or upgrade: helm -n app-staging status myapp-staging helm -n app-staging uninstall myapp-staging # Confirm cleanup kubectl -n app-staging get all Re-run the upgrade once relevant issues are fixed. ### Handling CRDs - CRDs often need to be applied separately from application resources. - Strategy: first apply CRD chart or manifest; then upgrade the app chart. If CRDs changed shape, perform a two-step deploy to avoid breaking existing CRs. ### Readiness Probe Issues If rollouts regularly time out, fix the probe (path, port, initialDelaySeconds) to match real startup behavior. Avoid simply raising timeouts as a first response. ### Postmortem Breadcrumbs Keep rendered.yaml , helm get manifest outputs, and kubectl events for failed runs. These accelerate root-cause analysis and prevent repeat incidents. ## Operations Checklist Use this list for day-to-day operation. Adjust names and timeouts to your environment. ### Preflight (per change) - Update values file with a pinned image tag and env-specific settings. - helm lint charts/myapp - helm template myapp charts/myapp -f values/staging.yaml > rendered.yaml - Peer review rendered.yaml for risky changes (ingress, securityContext, resource limits). ### Staging Deploy - helm upgrade --install myapp-staging charts/myapp -n app-staging -f values/staging.yaml --atomic --timeout 5m - kubectl -n app-staging rollout status deploy/myapp --timeout=120s - helm -n app-staging test myapp-staging --logs - Observe metrics and logs for at least one traffic slice if applicable. ### Promote to Production - Ensure the exact same chart version and image tag. - helm upgrade --install myapp charts/myapp -n app-prod -f values/production.yaml --atomic --timeout 10m - kubectl -n app-prod rollout status deploy/myapp --timeout=300s - helm -n app-prod test myapp --logs ### If Something Fails - Collect logs: kubectl describe , events, and helm get manifest . - Roll back quickly: helm rollback ... --wait --atomic - Open a follow-up to fix the root cause before the next promotion. ### Weekly Hygiene - Review helm history depth; prune if needed. - Validate values.schema.json covers new fields. - Confirm cluster and Helm versions remain within your tested range. ## Conclusion A reliable Helm CI/CD setup is built on small, verifiable steps. Keep versions consistent and environments clearly scoped. Fail fast with linting, schema checks, and templating before you ever touch the cluster. Use atomic upgrades, explicit timeouts, and test hooks to keep rollouts safe and observable. Prove success with concrete checks and keep artifacts for fast triage. Practice rollback and recovery so the team can execute confidently under pressure. Next steps: - Pilot the workflow on one service in staging with a pinned image tag and narrow scope. - Measure success criteria you care about (e.g., time to deploy, change failure rate). - Iterate on values.schema.json coverage and tests. - Roll the pattern out to additional services once you are satisfied with stability.