Apache Spark local lab setup with practical examples: practical implementation guide

Intro

A local Spark lab lets you learn fast, test ideas safely, and build repeatable habits before you touch shared systems. In this guide you will:

Install Spark and verify local mode.
Create a clean lab directory with data, logs, and checkpoints.
Run batch and streaming examples you can inspect.
Submit jobs programmatically and add simple tests.

By the end, you will have a workflow you can reuse for troubleshooting, experiments, and development.

Workflow Overview

Use a simple, repeatable flow that keeps changes small and visible:

Define a narrow goal and success criteria.
Prepare the lab folders and sample data.
Configure Spark defaults for consistent runs and logging.
Run code in local mode and inspect the Spark UI.
Capture results in versioned output folders.
Add a small test to guard core transforms.
Iterate in small steps, changing one thing at a time.

This separation of steps reduces rework and makes issues easier to isolate.

Local Lab Setup

Prerequisites

CPU: 4 cores
RAM: 8 GB minimum
Disk: 10 GB free
Java 11 or 17
Python 3.9+ if you use PySpark

Create the lab folders

mkdir -p ~/labs/spark/data/batch/input
mkdir -p ~/labs/spark/data/batch/output
mkdir -p ~/labs/spark/data/stream/input
mkdir -p ~/labs/spark/logs/events
mkdir -p ~/labs/spark/checkpoints

Install Spark (pre-built binaries)

Choose a Spark 3.5.x pre-built distribution for Hadoop 3.
Unpack it, for example to ~/tools/spark-3.5.1-bin-hadoop3.
Set environment variables:

macOS or Linux (bash/zsh):

export SPARK_HOME=~/tools/spark-3.5.1-bin-hadoop3
export PATH="$SPARK_HOME/bin:$PATH"
# Set JAVA_HOME to your installed JDK
# macOS example:
export JAVA_HOME=$(/usr/libexec/java_home -v 17)

Windows (PowerShell, paths will vary):

setx SPARK_HOME "C:\\tools\\spark-3.5.1-bin-hadoop3"
setx PATH "%SPARK_HOME%\\bin;%PATH%"
# Set JAVA_HOME to your installed JDK path as a user or system env var

Configure Spark defaults

Copy the template:

cp "$SPARK_HOME/conf/spark-defaults.conf.template" "$SPARK_HOME/conf/spark-defaults.conf"

Edit spark-defaults.conf and add:

spark.eventLog.enabled true
spark.eventLog.dir file:///home/youruser/labs/spark/logs/events
spark.sql.shuffle.partitions 4

Adjust the event log path for your OS and username.

Verify the install

spark-shell --version
pyspark --version

Run an interactive session and exit:

pyspark --master local[2]
# In the shell:
spark.range(5).show()
# Visit http://localhost:4040 to see the Spark UI while the shell is active.
exit()

Practical Examples

1) Batch aggregation: CSV to Parquet

Prepare a tiny CSV:

cat > ~/labs/spark/data/batch/input/customers.csv << 'EOF'
customer_id, name, country, spend
1, Alice, US,120.5
2, Bob, DE,95
3, Cara, US,250
4, Dan, FR,40
5, Eve, US,
EOF

Create batch_example.py:

# batch_example.py
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.functions import col
import argparse


def transform(df):
    clean = (
        df.filter(col("spend").isNotNull())
          .withColumn("spend_usd", col("spend").cast("double"))
    )
    agg = (
        clean.groupBy("country")
             .agg(
                 F.count("*").alias("customers"),
                 F.round(F.avg("spend_usd"), 2).alias("avg_spend")
             )
             .orderBy("country")
    )
    return agg


def main(input_path: str, output_path: str):
    spark = SparkSession.builder.appName("batch-agg").getOrCreate()
    df = (spark.read
               .option("header", "true")
               .option("inferSchema", "true")
               .csv(input_path))

    result = transform(df)
    (result.write
           .mode("overwrite")
           .parquet(output_path))

    result.show(truncate=False)
    spark.stop()


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--input", required=True)
    parser.add_argument("--output", required=True)
    args = parser.parse_args()
    main(args.input, args.output)

Run it with spark-submit:

cd ~/labs/spark
spark-submit \
  --master local[4] \
  --name batch-agg \
  batch_example.py \
  --input data/batch/input \
  --output data/batch/output/parquet

Inspect outputs:

ls -R data/batch/output/parquet

While the job runs, open the UI:

http://localhost:4040 for DAGs and stages
Check event logs in ~/labs/spark/logs/events

2) Structured Streaming: file source to console

Create stream_example.py:

# stream_example.py
from pyspark.sql import SparkSession
from pyspark.sql.functions import window, col, count

spark = (
    SparkSession.builder
    .appName("word-count-stream")
    .master("local[2]")
    .getOrCreate()
)

schema = "ts timestamp, word string"
stream = (
    spark.readStream
         .schema(schema)
         .json("data/stream/input")
)

counts = (
    stream.withWatermark("ts", "1 minute")
          .groupBy(window(col("ts"), "30 seconds"), col("word"))
          .agg(count("*").alias("n"))
)

query = (
    counts.writeStream
          .outputMode("append")
          .format("console")
          .option("truncate", "false")
          .trigger(processingTime="5 seconds")
          .start()
)

print("Streaming started. Append JSON lines to data/stream/input ...")
query.awaitTermination()

Start the stream:

cd ~/labs/spark
python3 stream_example.py

In another terminal, append events:

cd ~/labs/spark
cat > data/stream/input/batch1.json << 'EOF'
{"ts":"2024-01-01T10:00:00","word":"spark"}
{"ts":"2024-01-01T10:00:10","word":"spark"}
{"ts":"2024-01-01T10:00:20","word":"lab"}
EOF

You should see rolling counts in the console. Visit http://localhost:4040 to observe streaming queries and state operations while it is running.

3) Minimal test for the batch transform

Add test_batch_example.py:

# test_batch_example.py
import pytest
from pyspark.sql import SparkSession, Row
from batch_example import transform

@pytest.fixture(scope="session")
def spark():
    return (
        SparkSession.builder
        .appName("test-batch-agg")
        .master("local[2]")
        .getOrCreate()
    )

def test_transform_basic(spark):
    rows = [
        Row(customer_id=1, name="A", country="US", spend=100.0),
        Row(customer_id=2, name="B", country="US", spend=50.0),
        Row(customer_id=3, name="C", country="DE", spend=70.0),
        Row(customer_id=4, name="D", country="US", spend=None),
    ]
    df = spark.createDataFrame(rows)
    out = transform(df)
    got = { (r["country"], r["customers"], float(r["avg_spend"])) for r in out.collect() }
    assert ("DE", 1, 70.0) in got
    assert ("US", 2, 75.0) in got

Run the test:

pytest -q

Local Pilot Plan

Pick one task you can measure and inspect quickly:

Goal: Convert a small CSV to Parquet and compute per-country averages.
Scope: <= 10 MB of input data, 1 output folder, 1 transform function.
Success criteria:
Output Parquet exists with expected columns and row counts.
The Spark UI shows completed stages with no failures.
Runtime under 30 seconds on your machine.
Timebox: 1 to 2 hours, including setup and a basic test.

Why this works: it is narrow, measurable, and easy to inspect locally. You get fast feedback and can grow to more sources later (for example, add Kafka or HDFS once the local path is solid).

Troubleshooting and Tips

Java errors: Ensure java -version matches the JDK you expect (11 or 17). Set JAVA_HOME explicitly.
pyspark not found: Confirm SPARK_HOME/bin is on your PATH.
Spark UI not available: The UI runs only while a Spark app is active. Start a shell or job, then visit http://localhost:4040.
Port conflicts: If port 4040 is used, Spark will try 4041, 4042, etc. Check terminal output for the actual port.
Memory: For local runs, try --driver-memory 2g and adjust spark.sql.shuffle.partitions to a small value (for example 4).
File permissions: Keep your lab under your home directory to avoid permission issues.
Reproducibility: Fix Spark version, pin Python package versions, and keep sample data in version control.

Conclusion

You now have a safe local Spark lab, two practical examples, a minimal test, and a clear pilot plan. Keep iterations small, verify outputs and performance in the UI and logs, and only then add integrations like Kafka, HDFS, Airflow, or NiFi. This approach keeps learning fast and failures inexpensive while you build toward production-scale work.