## Introduction Apache Spark backup is not a single operation. Spark is a distributed compute engine; the data and metadata you must protect usually live outside Spark in file systems, object stores, catalogs, and external services. A reliable strategy distinguishes between: - Application recovery: your Spark code, dependencies, and configuration. - Data recovery: the datasets and table metadata Spark reads and writes. - Cluster recovery: the compute and platform configuration that runs Spark. This guide explains Apache Spark backup, restore, rollback, disaster recovery, and validation with concrete examples for HDFS and object storage, Hive Metastore and catalogs, table formats (Iceberg, Delta Lake, Hudi), and Structured Streaming checkpoints. It shows how to test restores safely and how to decide among rollback options. ## What must be backed up in a Spark environment?
ComponentWhat it containsIs it authoritative?Backup methodRecovery priority
Spark application source codeJobs, packages, testsYes (for logic)Version control with release tagsHigh
JAR and Python dependenciesBuilt artifacts, wheelsYes (for reproducibility)Artifact registry; lock filesHigh
spark-submit configurationSubmit args, Spark conf, job paramsYes (for behavior)Code repo and parameter storeHigh
Environment configurationSecrets refs, endpoints, IAM rolesYesIaC repo; parameter/secret storeHigh
HDFS or object-storage datasetsRaw, refined, curated dataYes (for data)Storage-level backups, replication, versioningCritical
Hive Metastore or catalog databaseTables, schemas, locationsYes (for metadata)Database dump/restore, snapshotsCritical
Iceberg/Delta/Hudi metadataSnapshots, transaction logsYes (for table integrity)Backup metadata and data files; controlled retentionCritical
Structured Streaming checkpointsOffsets, state store, commitsYes (for streaming continuity)Durable storage snapshots/copiesHigh
Spark event logsJob history for ops and auditsNo (supporting)File copies to durable storageMedium
Airflow or other orchestrationDAGs, schedules, variablesYes (for scheduling)VCS + export/snapshotsHigh
Kubernetes manifests or Helm valuesDeploy spec, images, quotasYes (for infra)GitOps repo, registryHigh
Secrets and credentialsKeys, tokens, passwordsYes and sensitiveEncrypted secret manager backups onlyHighest, secured
Never paste secrets into source repositories or backup scripts. Use a secret manager and back up its encrypted state per vendor guidance. ## Define recovery objectives before choosing a backup method - Recovery Point Objective (RPO): maximum tolerable data loss (time). Example: RPO 4 hours. - Recovery Time Objective (RTO): time to restore service. Example: RTO 2 hours. - Retention period: how long backups are kept (e.g., 90 days, 1 year). - Restore granularity: file, partition, table version, or full environment. - Cross-region or off-site recovery: protect against regional failures. - Immutable backups: prevent tampering and ransomware impact. - Encryption: in transit and at rest; rotate and back up keys securely. - Access controls: least privilege to read/restore backups. Example: a Spark pipeline that runs every 14 days on month-end data may set RPO = 24 hours (acceptable to re-run the last day of staging) and RTO = 4 hours. Retain curated table backups for 15 months for regulatory reporting and keep immutable off-site copies of table metadata and data files. ## Scenario 1: Backing up Spark application code and configuration A typical repo layout: spark-pipeline/ jobs/ config/ requirements.txt build.sbt Dockerfile kubernetes/ airflow/ tests/ Best practices: - Version control: trunk-based or GitFlow; protect main with CI checks. Use annotated release tags. - Dependency locking: use requirements.txt or poetry.lock for Python; SBT/Maven locks for Scala. Publish artifacts to a registry. - Artifact repositories: push JARs, wheels, and Docker images to a registry. Avoid mutable tags; prefer semver like 1.4.2. - Configuration separation: keep environment-specific config in separate files or parameter stores. Do not hardcode secrets. - Reproducible images: pin base image digests and Spark versions; document spark-submit args. Examples: # Tag a release git tag -a v1.4.2 -m "Month-end pipeline improvements" git push origin v1.4.2 # Create a reproducible source archive git archive --format=tar.gz --prefix=spark-pipeline-1.4.2/ v1.4.2 > spark-pipeline-1.4.2.tar.gz A Git repository is not a backup of the processed data. It only captures code and configuration. ## Scenario 2: Backing up data stored in HDFS HDFS replication guards against disk/node loss but is not a backup. Use DistCp to copy data to another HDFS cluster or to object storage. Inspect and plan: hdfs dfs -ls -h hdfs:///data/curated/sales/ hdfs dfs -count -h hdfs:///data/curated/sales/ hdfs dfs -mkdir -p hdfs:///backups/curated/sales/ Copy with DistCp: hadoop distcp \ hdfs:///data/curated/sales/ \ hdfs:///backups/curated/sales/ Verify counts and sizes: hdfs dfs -count -h hdfs:///data/curated/sales/ hdfs dfs -count -h hdfs:///backups/curated/sales/ Restore safely to a separate path first: hdfs dfs -mkdir -p hdfs:///restore/curated/sales/ hadoop distcp hdfs:///backups/curated/sales/ hdfs:///restore/curated/sales/ Validate with Spark before replacing production: # PySpark validation example src = "hdfs:///data/curated/sales/" restored = "hdfs:///restore/curated/sales/" df_src = spark.read.parquet(src) df_rst = spark.read.parquet(restored) assert df_src.count() == df_rst.count() assert df_src.schema.simpleString() == df_rst.schema.simpleString() Restoring into a separate path avoids overwriting good data if validation fails. ## Scenario 3: Backing up data in S3-compatible or cloud object storage Spark does not perform storage backups. Use storage-native capabilities: - Object versioning: keep previous versions to recover from deletions or overwrites. - Lifecycle policies: transition old data to colder tiers; expire when compliant. - Replication: cross-region or cross-account replication for disaster recovery. - Object Lock or immutability: write-once-read-many to block tampering. - Access controls: separate backup credentials and strong monitoring. Exact commands differ for AWS S3, Azure Data Lake Storage, Google Cloud Storage, and S3-compatible systems such as MinIO. At minimum, enable versioning, configure a lifecycle policy, and periodically verify that table data, transaction logs, and manifests are included in replicated or immutable sets. ## Scenario 4: Backing up a Hive Metastore or catalog database Copying Parquet files alone is not enough. You must preserve table definitions, schemas, partition metadata, and table locations so reads resolve correctly. Example with PostgreSQL-backed Metastore: Backup: pg_dump -h -U -d -Fc -f /backups/hms_$(date +%F).dump Restore to a recovery database: createdb -h -U hms_recovery pg_restore -h -U -d hms_recovery -c /backups/hms_YYYY-MM-DD.dump Validate entries: - Check that table locations reference the intended HDFS or object paths. - Run a smoke query via Spark SQL against the restored catalog and confirm partitions are discoverable. ## Scenario 5: Apache Iceberg, Delta Lake, and Hudi recovery Each table format separates data files from metadata and maintains a timeline of changes. - Iceberg: metadata.json files and snapshots reference manifests and data files. Retention is controlled by expire snapshots. - Delta Lake: _delta_log contains JSON and checkpoint Parquet files. History can be vacuumed away if retention is too aggressive. - Hudi: a timeline (commit, delta commit, compaction) controls versions; clean policies may remove old instants. Time travel helps rollback, but it is not a substitute for an independent backup. If you vacuum or expire metadata too aggressively, older versions become unrecoverable. Examples: - Delta Lake: spark.read.format("delta").option("versionAsOf", 17).load("/tables/sales_delta").show() - Iceberg: spark.read.format("iceberg").option("snapshot-id", "").load("prod.catalog.sales_iceberg").show() - Hudi: spark.read.format("hudi").option("as.of.instant", "20240101120000").load("/tables/sales_hudi").show() ## Scenario 6: Structured Streaming checkpoints A Structured Streaming checkpoint directory stores source offsets, commit logs, and state-store data. It must be on durable storage (e.g., HDFS or object storage). Deleting or reusing a checkpoint for a different query can cause reprocessing, skips, or state loss. Example: query = (spark.readStream.format("kafka") .option("kafka.bootstrap.servers", ":9092") .option("subscribe", "sales-events") .load() .selectExpr("CAST(value AS STRING) as json") .writeStream .format("parquet") .option("path", "s3a://datalake/curated/sales_events/") .option("checkpointLocation", "s3a://datalake/checkpoints/sales_events/") .outputMode("append") .start()) query.awaitTermination() Safe restarts reuse the same durable checkpoint with unchanged logic and schema. Copying a checkpoint does not guarantee recovery if code, schema, or source retention changed. ## Scenario 7: Restoring a failed Spark batch pipeline Assume a job reads raw Parquet, transforms it, and writes a curated table. - Stop dependent writes and pause consumers. - Identify the last known-good snapshot or table version. - Preserve the failed output under a quarantine path. - Restore data files or select a prior table version (time travel). - Restore catalog metadata if table definitions were corrupted. - Validate in a temporary location. - Compare counts, schemas, partitions, and business totals. - Switch consumers after validation. - Document what happened and what to improve. Validation examples: cur = spark.read.format("delta").load("/tables/sales_delta") prev = spark.read.format("delta").option("versionAsOf", 42).load("/tables/sales_delta") # Row count assert cur.count() == prev.count() # Schema assert cur.schema.json() == prev.schema.json() # Null checks from pyspark.sql.functions import col, sum as _sum nulls = cur.select([_sum(col(c).isNull().cast("int")).alias(c) for c in cur.columns]) nulls.show() # Duplicate-key checks pk = ["order_id"] cur.groupBy(pk).count().where("count > 1").show() # Business totals from pyspark.sql.functions import round cur.groupBy().sum("amount").select(round("sum(amount)", 2).alias("total")).show() # Sample comparison cur.orderBy(pk).limit(10).show(truncate=False) prev.orderBy(pk).limit(10).show(truncate=False) ## Scenario 8: Recovering a Structured Streaming job Kafka-to-lake example considerations: - Restart with the existing durable checkpoint to resume from stored offsets. - Kafka offset retention limits apply; if offsets have expired, replay may need to start from earliest available or a specific timestamp. - Ensure the sink is idempotent (e.g., MERGE with primary keys) or designed for exactly-once semantics within practical limits. - If the checkpoint is lost, create a new checkpoint and reprocess from a chosen offset boundary, coordinating with downstream consumers. ## Disaster recovery architecture A pragmatic design: - Primary Spark cluster (Kubernetes/YARN/VMs) created by automation. - HDFS or object storage for data and checkpoints. - Metastore or catalog database on a managed RDBMS. - Version-controlled application code and IaC. - Artifact and image registries. - Off-site, immutable backups of data and metadata. - Secondary environment that can be created on demand. - Monitoring and periodic restore tests. Text diagram: [Code+IaC] -> [CI/CD] -> [Artifact/Image Registry] -> [Primary Spark Cluster] | [HDFS/Object Storage] <-> [Catalog DB] | [Immutable Off-site Backups] | [Secondary Environment] ## Backup schedule example Asset Frequency Retention Storage location Validation method Owner Application code On change Forever Git repo Tag build, CI tests Data Eng JARs/images On release 2 years Artifact registry Pull and run smoke Platform HDFS datasets Daily DistCp 90 days DR HDFS cluster Count, size, checksum Data Ops Object-store datasets Continuous versioning 12 months Cross-region bucket Restore sample partitions Data Ops Catalog database Hourly snapshot 30 days Managed DB backups Restore to staging and query DBA Streaming checkpoints Hourly snapshot 14 days Durable storage Restart job in staging Data Eng Cluster config On change 1 year IaC repo Recreate cluster in lab Platform Event logs Daily 30 days Object storage Spark History UI check SRE ## Apache Spark rollback strategies - Application rollback: deploy previous code or image tag when logic is faulty. - Configuration rollback: revert Spark or job config parameters. - Dataset rollback: restore files or partitions from backup. - Table snapshot rollback: time travel to a prior version/snapshot. - Catalog restore: recover table definitions and locations from DB snapshot. - Full disaster recovery: fail over to secondary environment. When to use which: - Bad code release: application rollback. - Mis-tuned job: configuration rollback. - Corrupted output files: dataset or table snapshot rollback. - Dropped tables or wrong locations: catalog restore. - Regional outage: full disaster recovery. ## Restore validation checklist - Files present; counts and total size are plausible. - Checksums or etags match where supported. - Table schemas match expectations. - Partitions are discoverable and queryable. - Catalog entries resolve to the correct locations. - Record counts and key business aggregates are plausible. - Duplicate and null rates within tolerance. - Streaming checkpoints load; offsets and state are valid. - Downstream consumers can read restored outputs. - Permissions and ACLs are correct. - Monitoring and alerts function after restore. ## Common mistakes - Treating HDFS replication as backup. - Backing up code but not data, or data but not the catalog. - Storing checkpoints on ephemeral disks. - Overwriting production data during restore tests. - Never testing restores. - Keeping backups in the same failure domain. - Using mutable image tags such as latest. - Losing or not backing up encryption keys. - Expiring Iceberg snapshots or Delta history too aggressively. - Assuming time travel replaces disaster recovery. - Ignoring external systems (Kafka, RDBMS, Airflow, secrets management). ## Practical recovery drill Run this safely in a non-production environment: - Preparation: pick a medium table; note current version; ensure versioning and catalog backups are enabled. - Simulated failure: write a bad partition or overwrite the table with a small erroneous batch. - Restore: time travel to the prior version or copy files from backup to a recovery path; restore catalog to staging. - Validation: run counts, schema checks, aggregates; compare to pre-failure snapshot. - Rollback: atomically switch table pointer or replace partition with validated recovery data. - Documentation: record timings, steps, and gaps; update runbooks. - Success criteria: RTO and RPO met; downstream reads succeed; monitoring green. ## Conclusion Apache Spark compute is usually reproducible from code, images, and IaC. The parts that require deliberate protection are data, metadata, checkpoints, configuration, and artifacts. Do not conflate storage redundancy with backups, and do not rely on table time travel as a complete backup. Define explicit RPO and RTO targets, back up each authoritative layer with appropriate tooling, and prove your Apache Spark restore plan through regular, automated validation. A backup is only trustworthy after a successful restore test.