Apache Spark Basic Commands with Practical Examples

Introduction

Apache Spark is a unified analytics engine for large-scale data processing, and knowing its fundamental commands is essential for anyone operating Spark clusters or developing data pipelines. This guide provides a practical, command-focused walkthrough of common Spark operations: checking versions, inspecting your environment, making safe configuration changes, verifying cluster health, and handling failures. Each section includes specific commands, expected outputs, and tips to help you move from a problem to a verified solution.

Whether you are a developer, DevOps engineer, or part of a technical startup team, this article serves as both a tutorial and a handy reference—a cheat sheet for daily Spark operations. We'll prioritize operational safety: observe before changing, minimize the blast radius of any modification, use placeholders instead of real credentials, and always define a recovery path.

Version and Environment Inventory

Before running any Spark job or tweaking settings, know exactly what you're working with. This inventory phase is purely observational—no changes are made. Capture the version, deployment mode, and available resources.

Checking Spark Version

The most basic command is checking the Spark version. Run:

spark-submit --version

Expected output includes lines like:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.5.1
      /_/

If you're using PySpark, you can also check from a Python shell:

import pyspark
print(pyspark.__version__)

Inspecting the Deployment Topology

Understanding how Spark is deployed (standalone, YARN, Kubernetes) affects how you interact with it. Use the Spark UI or REST API. To get cluster information via REST, use:

curl -s http://<master-host>:8080/json/

This returns JSON with workers, applications, and more. Replace <master-host> with your actual Spark master address. If using YARN, the resource manager UI is typically at port 8088, and you can use yarn node -list to see nodes.

Listing Available Resources

Check how many cores and how much memory are available for your application. In standalone mode, use the Spark UI: http://<master-host>:8080. Look for the Workers section. For a command-line check, use spark-shell or pyspark to execute:

sc.statusTracker.getExecutorInfos

This shows active executors, their cores, and memory.

Prerequisites and Compatibility

Ensure your Java version is compatible. Spark 3.x requires Java 8/11/17. Check with java -version. If you're using Hadoop, verify compatibility with your HDFS version. Mismatched versions can cause cryptic errors.

Safe Configuration Path

Changing Spark configuration can be risky. Always follow a safe path: understand current settings, start with a minimal change, and verify the impact.

Viewing Current Configuration

To see all current Spark settings, use the Spark UI: http://<master-host>:8080 and navigate to the Environment tab for a running application. Or, from the command line, use:

spark-submit --help

To view specific configuration values, use spark-submit with --verbose (though it shows only some). Alternatively, use a simple Scala snippet:

spark.conf.get("spark.executor.memory")

Making a Configuration Change

Suppose you want to increase executor memory for a job. The safest way is to set it per job with spark-submit:

spark-submit \
  --class com.example.MyApp \
  --master yarn \
  --executor-memory 4g \
  --num-executors 10 \
  myapp.jar

Avoid changing cluster-wide defaults unless necessary. If you must, edit spark-defaults.conf in $SPARK_HOME/conf. Back up the file first:

cp spark-defaults.conf spark-defaults.conf.bak

Then make a single change, for example:

spark.executor.memory 4g

Verify the Change

After submitting a job with the new setting, confirm it took effect. In the Spark UI, go to the Executors tab and check the Memory column. Or, from the application log, look for a line like:

INFO MemoryStore: MemoryStore started with capacity 4.0 GB

Recovery Path

If the change causes performance degradation or failures, revert it. Restore the backup:

cp spark-defaults.conf.bak spark-defaults.conf

Then restart the affected services (if it's a master or worker config).

Verification and Diagnostics

Once you have a Spark cluster, you need to verify it's healthy and diagnose issues. Here are key commands and tools.

Checking Cluster Health

For standalone mode, use the REST API to see all workers and their states:

curl -s http://<master-host>:8080/json/ | jq '.workers'

Look for state: ALIVE. For YARN, use yarn node -list -all to see active nodes.

Running a Test Job

A quick way to verify the cluster is to run a simple Spark example. Spark includes a Pi calculation sample:

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode cluster \
  $SPARK_HOME/examples/jars/spark-examples_2.12-3.5.1.jar 10

Expected output includes lines like:

Pi is roughly 3.141439

This confirms resource allocation and job execution.

Monitoring Applications

Use the Spark UI for real-time monitoring. To see a list of applications, go to http://<master-host>:8080 and click on an application ID. Key metrics: Stages, Jobs, Storage, and Environment tabs. You can also use the REST API to fetch application status:

curl -s http://<master-host>:8080/api/v1/applications

Logging and Log Analysis

Logs are crucial for diagnostics. In YARN, get logs for a completed application with:

yarn logs -applicationId <application-id>

For standalone mode, logs are on workers in $SPARK_HOME/logs. Look for files like spark-<user>-org.apache.spark.deploy.worker-<host>.out. grep for errors:

grep -i "error" worker.log

Common Diagnostic Commands

• Check Spark shell connectivity: spark-shell (or pyspark) then run sc.version to confirm context is created.
• Check HDFS connectivity (if using HDFS): hdfs dfs -ls / to see if HDFS is reachable.
• Test network to shuffle: use curl -s http://<spark-ui-host>:8080 or check for port availability with nc -zv <host> 8080.

Failure Modes and Recovery

Failures happen. Knowing common failure modes and how to recover is vital.

Common Failures and Their Symptoms

• Out of Memory (OOM): Symptom: job fails with java.lang.OutOfMemoryError or executors killed. Look for logs like Container killed by YARN for exceeding memory limits.
• Recovery: Increase executor memory (--executor-memory) or reduce data per partition (increase parallelism with --conf spark.sql.shuffle.partitions=200).
• Connection Refused: Symptom: java.net.ConnectException when connecting to master. Check that the master process is running (jps should show Master and Worker).
• Recovery: Restart the master and workers. In standalone mode, run sbin/start-master.sh and sbin/start-slaves.sh.
• Serialization errors: Symptom: java.io.NotSerializableException. This happens when using functions that aren't serializable.
• Recovery: Make the class serializable or use foreachPartition with a connection object created per partition.
• Resource exhaustion: Symptom: application stuck in WAITING state. Check memory and core availability on the UI.
• Recovery: Free up resources, kill idle applications, or reduce the number of executors requested.

Diagnostic Command Example

For an OOM situation, you might run:

yarn logs -applicationId <app-id> | grep -i "OutOfMemory"

If you see Container killed, check the container logs:

curl -s http://<resourcemanager>:8088/proxy/<application-id>/logs

Recovery Paths

• Restart the application with higher memory:

spark-submit --executor-memory 8g --num-executors 5 ...

• Restart the Spark master/workers if the cluster is unstable:

$SPARK_HOME/sbin/stop-all.sh
$SPARK_HOME/sbin/start-all.sh

• Check for HDFS issues: use hdfs dfsadmin -report to see health.

Operations Checklist

Use this checklist when performing Spark operations.

Pre-Change Checklist

• [ ] Record the current version and deployment topology.
• [ ] Note the current configuration of the component you're changing.
• [ ] Understand the blast radius: which jobs and users might be affected?
• [ ] Back up any configuration files.

Change Execution

• [ ] Make the smallest change possible.
• [ ] Use placeholders for credentials and resource identifiers.
• [ ] Apply the change only to the target resource (e.g., one job, not cluster-wide).

Post-Change Verification

• [ ] Run a test job to verify the change works.
• [ ] Check the Spark UI for resource usage and success.
• [ ] Compare the output with expected results.

Recovery Checklist

• [ ] Know how to revert the change (restore backup, adjust settings).
• [ ] Document the recovery procedure before an incident.
• [ ] Communicate to team members what to watch for.

Conclusion

Mastering basic Spark commands is essential for smooth operations. By systematically checking versions, inspecting the environment, making safe configuration changes, verifying cluster health, and understanding failure modes, you can handle Spark with confidence. Always start with observation, validate each change, and keep a recovery plan ready. Use the commands and examples in this guide as a starting point for your own operational playbook.

As a next step, pick one low-risk verification task—like checking version or running a Pi job—and practice it. Record the current state, execute the command, and compare the result with the expected output. This hands-on practice builds the muscle memory needed for efficient Spark operations.

Remember, a reliable workflow makes failures visible, protects sensitive data, limits changes to the intended target, and defines recovery before an incident occurs.