Helm Backup and Restore with Practical Examples: A Hands-On Guide

Intro

Helm is the de facto package manager for Kubernetes, enabling developers and operators to deploy complex applications with a single command. However, as with any critical infrastructure, the state of your Helm releases can be lost due to accidental deletion, cluster failure, or misconfiguration. This guide provides practical, step-by-step instructions for backing up and restoring Helm releases, validating your backups, and planning for disaster recovery. By following these procedures, you can minimize downtime and ensure that your Kubernetes applications can be recovered quickly and reliably.

In this article, you will learn how to inventory your environment, implement a safe configuration path for backups, verify the integrity of your backups, handle common failure modes, and establish an operational checklist. Every major step includes concrete commands, configuration snippets, and expected outputs to help you execute these tasks in your own clusters.

Version and Environment Inventory

Before implementing any backup strategy, you must understand your current environment. This includes the versions of Helm and Kubernetes you are running, the storage backend used by Helm, and the locations of your release configurations and secrets.

Determine Helm and Kubernetes Versions

Run the following commands to check your Helm client and server versions:

helm version --short
kubectl version --short

Expected output (example):

v3.12.0+g3a31588
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

Knowing your Helm version is crucial because Helm 2 and Helm 3 have fundamentally different storage mechanisms. Helm 2 used Tiller and stored release information in ConfigMaps, while Helm 3 stores release information as Kubernetes Secrets by default in the release namespace. This guide assumes Helm 3.

Identify Storage Backend

By default, Helm 3 uses Secrets for storage. Verify this by checking the Helm configuration:

helm env

Look for the HELM_DRIVER variable. If not set, it defaults to secret. You can also inspect an existing release's storage object:

kubectl get secrets -n <namespace> | grep sh.helm.release

Example output:

sh.helm.release.v1.myapp.v1   helm.sh/release.v1   1      5d

Alternatively, if you have configured Helm to use ConfigMaps (by setting HELM_DRIVER=configmap), you would see ConfigMaps instead. Knowing the storage backend is essential for backing up the right resources.

Inventory Namespaces and Releases

List all Helm releases across all namespaces:

helm list --all-namespaces

Example output:

NAME    NAMESPACE   REVISION UPDATED                 STATUS   CHART           APP VERSION
myapp   default     1        2023-10-01 12:34:56.789 deployed myapp-1.2.3     1.0.0
web     production  3        2023-09-15 09:15:22.123 deployed nginx-1.5.0     1.23.0

Record the namespaces and release names. Also, identify any custom resource definitions (CRDs) or persistent volumes associated with your releases, as these may need separate backup strategies.

Prerequisites for Backup Tooling

To perform backups, you will need:

• kubectl configured with appropriate cluster access.
• helm client installed.
• Optional: jq for JSON processing, although not strictly required.
• For storage-level backups (e.g., PersistentVolume data), access to your cloud provider's snapshot tools or a storage API.

Safe Configuration Path

A safe backup strategy should be scoped to minimize impact and risk. Instead of backing up the entire cluster indiscriminately, focus on the Helm release metadata and the associated Kubernetes resources that are critical for recovery.

Backup Helm Release Metadata

Helm release metadata includes the chart version, values used, and the status of the release. Backing up this metadata allows you to reconstruct the release history and rollback if needed.

Option 1: Use helm get to Export Release Information

For each release, you can export its values and manifest:

# Export user-supplied values
helm get values <release-name> -n <namespace> -o yaml > release-values.yaml

# Export the full manifest as deployed
helm get manifest <release-name> -n <namespace> > release-manifest.yaml

Example command and output for values:

$ helm get values myapp -n default -o yaml
USER-SUPPLIED VALUES:
replicaCount: 3
image:
  repository: nginx
  tag: latest
service:
  type: LoadBalancer

Note that helm get values only returns user-supplied values, not defaults. If you need the effective values including defaults, use helm get values --all.

Option 2: Backup Helm Storage Objects Directly

Since Helm stores release information in Kubernetes Secrets (or ConfigMaps), you can back up these objects directly. For Secret-based storage:

kubectl get secrets -n <namespace> -l owner=helm -o yaml > helm-secrets-<namespace>.yaml

This command exports all Helm release secrets in the namespace. The label owner=helm is applied to Helm 3 release secrets by default. For ConfigMap storage, replace secrets with configmaps.

For multiple namespaces, loop through them:

for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl get secrets -n $ns -l owner=helm -o yaml > helm-secrets-$ns.yaml
done

Store these YAML files in a secure, version-controlled location.

Backup Associated Kubernetes Resources

Helm releases often create Deployments, Services, ConfigMaps, Secrets, and other resources. While these can be reconstructed from the chart and values, having a backup of the live resources can speed recovery and capture any manual changes.

Use kubectl to export resources labeled with the release:

# Get all resources for a release using the release label
kubectl get all -n <namespace> -l app.kubernetes.io/managed-by=Helm -o yaml > release-resources.yaml

Note: The label app.kubernetes.io/managed-by=Helm is commonly added by Helm 3 charts, but not all charts include it. Check the labels on your resources and adjust accordingly. Alternatively, you can capture all resources in the namespace, but that may include non-Helm resources.

Backing Up CRDs

If your charts install or rely on Custom Resource Definitions (CRDs), back them up separately:

kubectl get crds -o yaml > crds.yaml

CRDs are cluster-scoped and not tied to a namespace, so a single export covers all.

Backup Persistent Volume Data

Helm backups typically do not include data stored in PersistentVolumes. You must implement a separate backup strategy for persistent data, such as using your cloud provider's snapshot capability, Velero, or a database-specific backup tool. For demonstration, here is an example using a snapshot for a hypothetical cloud provider:

# Example for a cloud provider CLI (hypothetical)
cloud-provider snapshot create --volume-id <pv-id> --snapshot-name myapp-snapshot-$(date +%Y%m%d)

This command creates a snapshot of the PersistentVolume. The exact command depends on your environment.

Automation and Scheduling

For regular backups, schedule the export commands using cron jobs or a CI/CD pipeline. Here is an example of a simple shell script that backs up all Helm releases:

#!/bin/bash
BACKUP_DIR="/backups/helm/$(date +%Y%m%d_%H%M%S)"
mkdir -p $BACKUP_DIR
for ns in $(kubectl get ns -o jsonpath='{.items[*].metadata.name}'); do
  kubectl get secrets -n $ns -l owner=helm -o yaml > $BACKUP_DIR/helm-secrets-$ns.yaml
done
# Optionally upload to remote storage
# aws s3 cp $BACKUP_DIR s3://my-helm-backups/ --recursive

Schedule this script via a cron job on a management machine or a Kubernetes CronJob if you prefer to run it inside the cluster.

Verification and Diagnostics

After creating backups, you must verify that they are complete and usable. This section provides checks to ensure your backups are valid and that you can restore from them when needed.

Validate Backup Files

For YAML exports, ensure the files are syntactically correct and contain expected resources. Use kubectl apply --dry-run=client to validate without applying:

kubectl apply --dry-run=client -f helm-secrets-default.yaml

Expected output if valid:

secret/sh.helm.release.v1.myapp.v1 created (dry run)

If the file is malformed, you will see syntax errors.

For helm get output, verify that the values and manifest are non-empty and parse correctly:

helm get values myapp -n default -o yaml | yq eval '.' -

You can also compare the exported values to a known good set.

Test Restore in a Sandbox

The best verification is to perform a test restore in a separate namespace or cluster. This ensures that the backup contains all necessary information to recreate the release.

Restore from Secret Backup

If you backed up the release Secrets, you can restore them by applying the YAML to the target cluster. However, simply applying the Secret may not register the release with Helm's storage correctly because Helm uses specific annotations and labels. A safer approach is to use helm install with the exported values and the same chart version.

• Export the chart version used:

helm show chart <release-name> -n <namespace> | grep version

• Install the chart with the same version and exported values:

helm install <release-name> <chart-repo>/<chart-name> --version <chart-version> --values release-values.yaml -n <namespace>

Example:

helm install myapp myrepo/myapp --version 1.2.3 --values release-values.yaml -n default

Expected output:

NAME: myapp
LAST DEPLOYED: Mon Oct  2 10:00:00 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1

Restore from Storage Object Directly (Advanced)

If you prefer to restore the exact release metadata including revision history, you can apply the backed-up Secrets and then inform Helm. However, this is not recommended for most users because Helm may not recognize the release. For completeness, here is a method:

• Apply the backed-up Secret:

kubectl apply -f helm-secrets-default.yaml

• Check if Helm sees the release:

helm list -n default

If the release appears, great. If not, you may need to manually adjust labels and annotations.

Diagnose Backup Coverage

Ensure that all releases are backed up. Compare the list of releases from helm list --all-namespaces with the backup files. Write a simple script to check for missing namespaces or releases.

For example, a quick check:

# Compare release list to backup files
helm list --all-namespaces -o json | jq -r '.[].name' | sort > /tmp/releases.txt
ls helm-secrets-*.yaml | sed 's/helm-secrets-//;s/\.yaml//' | sort > /tmp/backups.txt
diff /tmp/releases.txt /tmp/backups.txt

If diff shows no output, all releases are accounted for.

Verify Persistent Volume Snapshots

For PV snapshots, verify that the snapshot was created successfully and can be used to restore a volume. For example, using a cloud CLI (hypothetical):

cloud-provider snapshot list --volume-id <pv-id>

Expected output shows the snapshot with status available.

If using Velero or a similar tool, check its logs and backup status to ensure PV data is included.

Failure Modes and Recovery

Despite best efforts, failures can occur. Understanding common failure modes and having a recovery plan is essential.

Common Failure Scenarios

• Accidental deletion of a Helm release: If you run helm uninstall, the release history is removed. Recovery requires reinstallation from backup.
• Corrupted or missing release Secret: If the Secret is deleted, Helm loses track of the release, but the deployed resources remain. You may need to recreate the Secret or adopt existing resources.
• Cluster failure: If the entire cluster is lost, you must restore from off-cluster backups.
• Persistent data loss: Helm metadata backups do not protect application data; if PV data is lost, you need snapshot restores.

Rollback and Recovery Procedures

Scenario 1: Accidental helm uninstall

If you have a backup of the release values and chart version, reinstall:

helm install <release-name> <chart> --version <version> --values release-values.yaml -n <namespace>

This creates a new release with the same name. Note that revision history is reset; if you had multiple revisions, you lose them unless you backed up the storage objects.

To retain revision history, avoid uninstalling and prefer helm rollback to a previous revision. However, if uninstall already happened, reinstallation is the only option.

Scenario 2: Missing Release Secret

If the Secret is deleted but the resources are still running, you can use helm adopt (available in Helm 3.13+) to adopt existing resources into a new release. For older versions, you can recreate the Secret manually. Example with helm adopt:

helm adopt myapp -n default --kube-context mycluster

helm adopt queries the cluster for resources matching the release name and labels, then creates the necessary release metadata.

If helm adopt is unavailable, you can create a new release with helm install using the existing resources, but you may need to annotate them to avoid conflicts.

Scenario 3: Full Cluster Recovery

In a disaster scenario, follow a documented runbook. Key steps:

1. Provision a new Kubernetes cluster.
2. Restore CRDs first: kubectl apply -f crds.yaml.
3. Restore persistent volumes from snapshots.
4. Reinstall Helm releases using backed-up values and chart versions.
5. Verify application health.

Testing Recovery

Regularly test recovery procedures to ensure they work. Schedule a quarterly disaster recovery drill where you simulate a failure and restore from backups into a staging cluster.

Common Pitfalls and How to Avoid Them

Even with a solid backup plan, mistakes happen. Here are some frequent pitfalls in Helm backup and restore processes, why they occur, and how to avoid or recover from them.

1. Not Backing Up the Right Storage Backend

Why it happens: Helm 3 defaults to Secrets, but some clusters are configured to use ConfigMaps via HELM_DRIVER. If you assume Secrets and your cluster uses ConfigMaps, your backup will be empty.

How to avoid: Always run helm env to confirm the storage driver before running backup commands. Adjust your backup commands accordingly.

Recovery: If you discover the error after a failure, check the current driver and run the correct backup command. Also, ensure your backup scripts detect the driver dynamically.

2. Omitting PersistentVolume Data

Why it happens: Many operators focus only on Helm metadata and forget that application data lives outside the release objects. A restore that brings back the deployment but not the database data is incomplete.

How to avoid: Integrate PV backups into your strategy from the start. Use cloud snapshots, Velero, or database-specific tools, and include PV restore steps in your runbook.

Recovery: If you have no PV backups, you may need to rely on application-level replication or recent exports. If data is corrupted, you might need to restore from an older snapshot and replay logs.

3. Relying Solely on helm get values Without Defaults

Why it happens: helm get values only returns user-supplied values, not chart defaults. If you restore using only these values, you may lose customizations that came from defaults, values.yaml in the chart, or previous upgrades.

How to avoid: Use helm get values --all to capture the effective values, or better, use the original values file stored in your Git repository.

Recovery: If you only have user-supplied values, you can still restore by overriding defaults as needed. Review the chart's default values.yaml to identify any critical settings that need to be set.

4. Not Testing Restores Regularly

Why it happens: Backups may be created but never tested. A backup that cannot be restored is useless.

How to avoid: Schedule regular restore tests in a staging namespace or cluster. Automate the test to run after each backup cycle.

Recovery: If a test fails, diagnose immediately: check for missing files, incorrect syntax, or version mismatches. Fix the backup process and re-test before considering the system reliable.

Operations Checklist

Use this checklist to ensure consistent backup and restore operations. For each item, a single owner is identified, and the review frequency is specified.

Conclusion

Backing up and restoring Helm releases is a critical operational task that, when done properly, protects your Kubernetes applications from data loss and downtime. By following the steps in this guide, you have established a solid foundation: you inventoried your environment, implemented a safe and scoped backup configuration, verified the integrity of your backups, and prepared for failure modes with recovery procedures.

Your next steps should be:

• Automate the backup process using the provided scripts.
• Set up off-cluster storage for backup files.
• Perform a test restore in a sandbox to validate your backups.
• Schedule regular disaster recovery drills.

With these practices in place, you can confidently operate Helm in production, knowing that your releases and their configurations are safe and recoverable.