## Intro

A local Spark lab lets you learn fast, test ideas safely, and build repeatable habits before you touch shared systems. In this guide you will:

- Install Spark and verify local mode.

- Create a clean lab directory with data, logs, and checkpoints.

- Run batch and streaming examples you can inspect.

- Submit jobs programmatically and add a minimal test.

By the end you will have a workflow you can reuse for troubleshooting, experiments, and development.

## Workflow Overview

Use a simple, repeatable flow that keeps changes small and visible:

- Define a narrow goal and success criteria.

- Prepare the lab folders and sample data.

- Configure Spark defaults for consistent runs and logging.

- Run code in local mode and inspect the Spark UI.

- Capture results in versioned output folders.

- Add a small test to guard core transforms.

- Iterate in small steps, changing one thing at a time.

This separation of steps reduces rework and makes issues easier to isolate.

## Local Lab Setup

### Prerequisites

- CPU: 4 cores

- RAM: 8 GB minimum

- Disk: 10 GB free

- Java 11 or 17 (verify with java -version )

- Python 3.9+ if you use PySpark

### Create the lab folders

mkdir -p ~/labs/spark/data/batch/input
mkdir -p ~/labs/spark/data/batch/output
mkdir -p ~/labs/spark/data/stream/input
mkdir -p ~/labs/spark/logs/events
mkdir -p ~/labs/spark/checkpoints 

### Install Spark (pre‑built binaries)

- Download Spark 3.5.1 pre‑built for Hadoop 3 from the official archive.

- Unpack it, for example to ~/tools/spark-3.5.1-bin-hadoop3 .

- Set environment variables.

 macOS / Linux (bash/zsh)

export SPARK_HOME=~/tools/spark-3.5.1-bin-hadoop3
export PATH="$SPARK_HOME/bin:$PATH"
# Set JAVA_HOME to your installed JDK
# macOS example:
export JAVA_HOME=$(/usr/libexec/java_home -v 17) 
 Windows (PowerShell)

setx SPARK_HOME "C:\tools\spark-3.5.1-bin-hadoop3"
setx PATH "%SPARK_HOME%\bin;%PATH%"
# Set JAVA_HOME to your installed JDK path as a user or system env var 

### Configure Spark defaults

 Copy the template and edit the defaults file:

cp "$SPARK_HOME/conf/spark-defaults.conf.template" "$SPARK_HOME/conf/spark-defaults.conf" 
 Add the following lines to spark-defaults.conf (replace youruser with your actual username):

spark.eventLog.enabled true
spark.eventLog.dir file:///home/youruser/labs/spark/logs/events
spark.sql.shuffle.partitions 4 

### Verify the install

 spark-shell --version
pyspark --version 
 Start an interactive PySpark session and run a quick sanity check:

pyspark --master local[2]
# In the shell:
spark.range(5).show()
# Visit http://localhost:4040 to see the Spark UI while the shell is active.
exit() 

## Practical Examples

### 1) Batch aggregation: CSV to Parquet

 Prepare a tiny CSV with a missing value in the last row:

cat > ~/labs/spark/data/batch/input/customers.csv << 'EOF'
customer_id,name,country,spend
1,Alice,US,120.5
2,Bob,DE,95
3,Cara,US,250
4,Dan,FR,40
5,Eve,US,
EOF 
 Create batch_example.py :

# batch_example.py
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.functions import col
import argparse

def transform(df):
 clean = (
 df.filter(col("spend\)).isNotNull())
 .withColumn("spend_usd", col("spend\)).cast("double\
 )
 )
 agg = (
 clean.groupBy("country\
 .agg(
 F.count("\*\(\).alias("customers\
 F.round(F.avg("spend_usd\
 2).alias("avg_spend\))
 )
 .orderBy("country\)
 )
 return agg

def main(input_path: str, output_path: str):
 spark = SparkSession.builder.appName("batch-agg\)).getOrCreate()
 df = (spark.read
 .option("header", "true\)
 .option("inferSchema", "true\)
 .csv(input_path))

 result = transform(df)
 (result.write
 .mode("overwrite\)
 .parquet(output_path))

 result.show(truncate=False)
 spark.stop()

if __name__ == "__main__":
 parser = argparse.ArgumentParser()
 parser.add_argument("--input", required=True)
 parser.add_argument("--output", required=True)
 args = parser.parse_args()
 main(args.input, args.output) 
 Run it with spark-submit :

cd ~/labs/spark
spark-submit \
 --master local[4] \
 --name batch-agg \
 batch_example.py \
 --input data/batch/input \
 --output data/batch/output/parquet 
 Inspect the output:

ls -R data/batch/output/parquet 
 While the job runs, open the Spark UI at http://localhost:4040 to view DAGs and stages, and check the event logs in ~/labs/spark/logs/events .

### 2) Structured Streaming: file source to console

Create stream_example.py :

# stream_example.py
from pyspark.sql import SparkSession
from pyspark.sql.functions import window, col, count

spark = (\n SparkSession.builder\n .appName("word-count-stream\)\n .master("local[2]\n .getOrCreate()\)

schema = "ts timestamp, word string"
stream = (\n spark.readStream\n .schema(schema)\n .json("data/stream/input\)
)

counts = (\n stream.withWatermark("ts", "1 minute\)\n .groupBy(window(col("ts\)), "30 seconds\)), col("word\)\n .agg(count("\*\(\).alias("n\))
)

query = (\n counts.writeStream\n .outputMode("append\)\n .format("console\)\n .option("truncate", "false\)\n .trigger(processingTime="5 seconds\)\n .start()\)

print("Streaming started. Append JSON lines to data/stream/input ...\)
query.awaitTermination() 
 Start the streaming job:

cd ~/labs/spark
python3 stream_example.py 
 In another terminal, append a few events:

cd ~/labs/spark
cat > data/stream/input/batch1.json << 'EOF'
{"ts":"2024-01-01T10:00:00","word":"spark"}
{"ts":"2024-01-01T10:00:10","word":"spark"}
{"ts":"2024-01-01T10:00:20","word":"lab"}
EOF 
 You should see rolling counts in the console. Visit http://localhost:4040 to observe streaming queries and state operations while it is running.

### 3) Minimal test for the batch transform

Add test_batch_example.py :

# test_batch_example.py
import pytest
from pyspark.sql import SparkSession, Row
from batch_example import transform

@pytest.fixture(scope="session\)
def spark():
 return (\n SparkSession.builder\n .appName("test-batch-agg\)\n .master("local[2]\n .getOrCreate()\)

def test_transform_basic(spark):
 rows = [
 Row(customer_id=1, name="A", country="US", spend=100.0),
 Row(customer_id=2, name="B", country="US", spend=50.0),
 Row(customer_id=3, name="C", country="DE", spend=70.0),
 Row(customer_id=4, name="D", country="US", spend=None),
 ]
 df = spark.createDataFrame(rows)
 out = transform(df)
 got = {(r["country"], r["customers"], float(r["avg_spend\))) for r in out.collect()}
 assert ("DE", 1, 70.0) in got
 assert ("US", 2, 75.0) in got 
 Run the test:

pytest -q 

## Local Pilot Plan

 Pick one task you can measure and inspect quickly:

- Goal : Convert a small CSV to Parquet and compute per‑country averages.

- Scope : ≤ 10 MB of input data, one output folder, one transform function.

- Success criteria :

- Output Parquet exists with expected columns ( country , customers , avg_spend ) and row count matching the number of distinct countries.

- The Spark UI shows completed stages with no failures.

- Runtime under 30 seconds on your machine.

- Timebox : 1 to 2 hours, including setup and a basic test.

Why this works: it is narrow, measurable, and easy to inspect locally. You get fast feedback and can grow to more sources later (for example, add Kafka or HDFS once the local path is solid).

## Troubleshooting and Tips

- Java errors : Ensure java -version matches the JDK you expect (11 or 17). Set JAVA_HOME explicitly.

- pyspark not found : Confirm $SPARK_HOME/bin is on your PATH .

- Spark UI not available : The UI runs only while a Spark application is active. Start a shell or job, then visit http://localhost:4040 .

- Port conflicts : If port 4040 is used, Spark will try 4041, 4042, etc. Check terminal output for the actual port.

- Memory : For local runs, try --driver-memory 2g --executor-memory 2g and keep spark.sql.shuffle.partitions small (for example 4).

- File permissions : Keep your lab under your home directory to avoid permission issues.

- Reproducibility : Fix Spark version, pin Python package versions (e.g., via requirements.txt ), and keep sample data in version control.

Related topics to explore next:

- Kafka: switch the streaming file source to a Kafka source once you have a broker.

- HDFS: point input and output to HDFS for larger tests.

- Apache Airflow or NiFi: orchestrate jobs after your local pilot is stable.

## Conclusion

You now have a safe local Spark lab, two practical examples, a minimal test, and a clear pilot plan. Keep iterations small, verify outputs and performance in the UI and logs, and only then add integrations like Kafka, HDFS, Airflow, or NiFi. This approach keeps learning fast and failures inexpensive while you build toward production‑scale work.