Apache Spark production operations checklist with practical examples: practical implementation guide

Intro

Running Apache Spark in production is straightforward when you standardize the steps that matter: configuration baselines, monitoring, reliability, storage, security, tuning, maintenance, and upgrades. This guide gives developers, DevOps consultants, and startup teams a concise, practical checklist with examples you can apply today.

Use it to reduce incidents, shorten recovery time, and keep costs in check while meeting SLAs for batch and streaming jobs across Kafka, HDFS, object storage, and schedulers like Airflow or NiFi.

Workflow Overview

Follow this end-to-end flow for dependable Spark operations:

1. Plan

• Define SLOs: job latency, data lag, data quality thresholds, and cost targets.
• Map dependencies: data sources (Kafka, HDFS, S3), sinks (Parquet, Delta, JDBC), catalogs, and schedulers.

1. Configure

• Set cluster and Spark defaults once, then override only by exception.
• Establish naming, logging, and checkpoint directory conventions.

1. Deploy and validate

• Dry-run on sample data, then shadow production with read-only validation.
• Gate promotion on success metrics and error budget.

1. Monitor and alert

• Instrument metrics, event logs, and logs. Wire alerts to leading indicators.

1. Maintain and back up

• Rotate logs, compact data, clean checkpoints, and back up catalogs.

1. Upgrade safely

• Test on staging with realistic data, keep a rollback plan and config diff.

Configuration Baselines

Lock in a safe default profile before scaling or onboarding teams.

Baseline spark-defaults.conf (tune for your workload size):

# Core execution
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.sql.shuffle.partitions=400                # set based on data size
spark.sql.adaptive.enabled=true                 # AQE
spark.sql.adaptive.coalescePartitions.enabled=true
spark.sql.adaptive.skewJoin.enabled=true

# Resource elasticity
spark.dynamicAllocation.enabled=true
spark.shuffle.service.enabled=true
spark.dynamicAllocation.minExecutors=2
spark.dynamicAllocation.maxExecutors=100        # cap by budget and SLA
spark.dynamicAllocation.initialExecutors=4

# Memory and CPU (adjust per workload)
spark.executor.instances=10
spark.executor.cores=4
spark.executor.memory=8g
spark.driver.memory=4g
spark.memory.fraction=0.6

# Reliability and timeouts
spark.network.timeout=600s
spark.sql.broadcastTimeout=300
spark.task.maxFailures=4
spark.speculation=true                          # handle slow tasks

# I/O and compression
spark.sql.parquet.compression.codec=zstd
spark.sql.parquet.filterPushdown=true
spark.sql.files.maxPartitionBytes=134217728     # 128 MB

Hadoop and object store tips (in spark-defaults or core-site.xml):

# S3A tuning examples (if using S3-compatible storage)
spark.hadoop.fs.s3a.fast.upload=true
spark.hadoop.fs.s3a.connection.maximum=200
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2

Conventions that prevent errors:

• Use separate, per-application checkpoint and staging directories.
• Tag applications with spark.app.name that encodes owner, purpose, and env.
• Pin connector versions compatible with your Spark and Scala versions.

Monitoring and Alerting

A minimal, effective telemetry stack includes metrics, event logs, and logs.

Metrics

• Enable Spark metrics system and expose a sink you can scrape (for example Prometheus JMX or servlet). Track:
• Job duration, failed stages, task retries
• Executor counts, lost executors, GC time, JVM heap usage
• Shuffle read/write, disk spill, storage memory
• Structured streaming: input rows, processed rows, offsets, state store metrics

Example metrics config snippet (metrics.properties):

*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
master.sources.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.sources.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.sources.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.sources.jvm.class=org.apache.spark.metrics.source.JvmSource

Event logs and UI

• Enable event logging to a durable store for postmortems.
• Keep a retention policy and index event logs for searchability.

Logging

• Set INFO for production, DEBUG only for targeted troubleshooting.
• Redact secrets (tokens, keys) and PII in logs.

Alert ideas

• Batch: job duration p95 exceeds SLO; stage retry rate > N%; shuffle spill ratio rising.
• Streaming: input lag grows for M minutes; checkpoint write failures; processing time spikes.
• Infra: lost executors > threshold; driver GC time > 20%; disk full on shuffle volumes.

Job Reliability and Scheduling

Make jobs restartable and idempotent.

Batch patterns

• Input partitioning: define stable boundaries (by date, hour, or id ranges).
• Idempotent writes: use overwrite by partition or merge semantics.
• Retries: allow Spark task retries, but cap them to avoid retry storms.

Structured Streaming checklist

• Always set a durable checkpointLocation.
• Prefer exactly-once sinks when available; otherwise design for at-least-once with dedupe.
• Control load with trigger intervals and backpressure strategies.

Example: Kafka to Parquet with checkpointing (Scala):

val df = spark.readStream
  .format("kafka")
  .option("kafka.bootstrap.servers", "broker:9092")
  .option("subscribe", "events")
  .option("startingOffsets", "latest")
  .load()
  .selectExpr("CAST(value AS STRING) as json")

val parsed = spark.read.json(df.select("json").as[String])

parsed.writeStream
  .format("parquet")
  .option("path", "/data/events_parquet")
  .option("checkpointLocation", "/chk/events_parquet")
  .outputMode("append")
  .start()

Scheduling integration example (Airflow Spark submit operator concept):

• Define retries with exponential backoff.
• Emit metrics and send on-failure alerts.
• Enforce SLAs per DAG task and surface them on dashboards.

Data Storage and I/O

Formats and partitioning

• Use columnar formats (Parquet) with predicate pushdown and projection.
• Partition by low-cardinality, high-selectivity columns (date, hour, country).
• Avoid over-partitioning; target 128 MB to 512 MB output file sizes.

Small files control

• For batch: write fewer, larger files using coalesce or AQE coalescing.
• Periodic compaction job: read small files, rewrite to target size.

Shuffle and joins

• Reduce shuffle by filtering and selecting early.
• Broadcast small dimension tables; tune thresholds:

spark.sql.autoBroadcastJoinThreshold=104857600   # 100 MB

Throughput tuning

• Increase parallelism by adjusting input splits and partitions.
• For S3-like stores, increase connections and multipart size cautiously.

Security and Governance

• Network and transport: enable TLS for Spark UI, RPC, and shuffle if supported by your platform.
• AuthN and AuthZ: use Kerberos or IAM roles; avoid static keys on disk.
• Secrets: mount via secure providers; never log them.
• Data: encrypt at rest; restrict bucket or HDFS paths by role.
• Audit: collect access logs for data and metastore operations.
• Log redaction: sanitize tokens, emails, and IDs in logs.

Cost and Performance Tuning

Executor sizing

• Start with 4 cores and 6-8 GB per executor; adjust based on spill and GC.
• Keep enough parallelism to saturate the cluster without thrashing.

Adaptive Query Execution (AQE)

spark.sql.adaptive.enabled=true
spark.sql.adaptive.skewJoin.enabled=true

• Lets Spark coalesce partitions and mitigate skew at runtime.

Partition counts

• Rule of thumb: target 1-3 tasks per CPU core; validate with stage timelines.

Skew mitigation

• Salt skewed keys or use AQE skew join; verify via Spark UI task time heatmaps.

Caching

• Cache only when reused multiple times and memory is sufficient. Prefer checkpoint for long lineage in streaming.

Dynamic allocation

• Set min, max, and idle timeouts to match SLA and cost objectives.

Spot or preemptible instances (if applicable)

• Use with dynamic allocation and test loss of executors; ensure checkpointed progress.

Maintenance and Backups

State and logs

• Checkpoints: do not delete active checkpoints; clean only after a full reset.
• Event logs: rotate and archive with lifecycle policies.
• Shuffle disks: monitor space and wear; alert well before full.

Catalog and metadata

• Back up the Hive metastore or cloud catalog on a schedule.
• Version and back up spark-defaults.conf, log configs, and job definitions.

Data hygiene

• Compaction: run regular jobs to merge small files.
• Vacuum: remove old snapshots if your table format supports it.

Disaster readiness

• Document recovery runbooks. Test restores quarterly.

Upgrades and Rollbacks

Pre-upgrade checklist

• Verify compatibility matrix: Spark, Scala, Java, connectors, Hadoop libs.
• Review SQL behavior changes and config defaults that may affect plans.
• Rebuild user-defined functions if binaries or ABIs changed.

Staged rollout

• Run canary jobs with production-like data and inputs.
• Compare metrics to baseline: duration, spill, shuffle, GC, cost.

Rollback plan

• Keep previous images and configs ready.
• Automate rollback and data path switches when possible.

Common Production Pitfalls

• Data skew
• Symptom: a few tasks run much longer.
• Fix: enable AQE skew join, salt keys, or pre-aggregate.

• Small files flood
• Symptom: too many tiny Parquet files; slow reads and high metadata cost.
• Fix: coalesce on write, enable AQE coalesce, add compaction jobs.

• Missing streaming checkpoints
• Symptom: duplicates or reprocessing after restarts.
• Fix: set checkpointLocation and use idempotent sinks.

• Driver overload
• Symptom: driver OOM or long GC pauses.
• Fix: reduce collect() and large driver-side operations; increase driver memory.

• Overusing Python UDFs
• Symptom: slow stages due to serialization overhead.
• Fix: use built-in Spark SQL functions or pandas UDFs when appropriate.

• Mis-sized shuffle partitions
• Symptom: either underutilized cluster or excessive overhead.
• Fix: tune spark.sql.shuffle.partitions; rely on AQE to coalesce.

Local Pilot Plan

Start small, measurable, and locally inspectable before production.

Scope

• One representative batch job and one short-lived streaming job.
• Data volume: 10-50 GB for batch; 1-5 partitions for streaming input.

Environment

• Run Spark in local mode for first validation:

spark.master=local[*]

Success metrics

• Batch: p95 runtime within target on sample; 0 failed stages; output file sizes near 128-512 MB.
• Streaming: processing time per trigger stable; lag under threshold; no checkpoint errors for 24 hours.

Failure drills

• Kill an executor or simulate network hiccups; confirm retries and idempotency.
• Corrupt one input file; verify job isolates and reports failure without cascading.

Promotion checklist

• Configs pinned and versioned; metrics and alerts live; runbooks updated.
• Rollback steps tested; prior version ready to redeploy.

Conclusion

A reliable Apache Spark production setup starts with clear baselines, actionable monitoring, and restartable jobs. Add storage and security guardrails, tune for cost and performance, and practice maintenance and backup routines. Upgrade in stages with a rollback plan, and begin with a narrow pilot you can measure and inspect locally. Apply this checklist, track a small set of leading indicators, and iterate after each release to keep Spark operations predictable and cost effective.