## Intro Apache Spark networking issues can stop a job before it starts, or worse, cause intermittent failures that are hard to reproduce. This guide provides practical, step-by-step troubleshooting for networking problems in Apache Spark. It is written for developers, DevOps engineers, and technical teams who operate Spark clusters and need to move from an observed problem to a verified result. We cover Apache Spark DNS resolution, common ports, connectivity checks, and network troubleshooting commands. Each section includes concrete commands with expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. No real credentials, tokens, or production identifiers are used. All examples are for a Spark 3.5.0 standalone cluster on Linux, but the principles apply to Spark on YARN, Kubernetes, or cloud services. ## Version and Environment Inventory Before troubleshooting, gather the exact version and topology. This avoids applying solutions meant for a different Spark release or deployment mode. Run these read-only commands on the driver and executors: # On the driver node spark-submit --version # Expected output (truncated): # Welcome to # ____ __ # / __/__ ___ _____/ /__ # _\ \/ _ \/ _ `/ __/ '_/ # /___/ .__/\_,_/_/ /_/\_\ version 3.5.0 # /_/ # Check Java version (required: Java 8/11/17 for Spark 3.5.0) java -version # Expected output: # openjdk version "11.0.20" 2023-07-18 # List all Spark processes and their ports ps -ef | grep -i spark # Example output (partial): # spark 1234 1 0 10:00 ? 00:00:00 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Xmx1g org.apache.spark.deploy.master.Master --host master.example.com --port 7077 --webui-port 8080 # spark 5678 1 0 10:01 ? 00:00:00 ... org.apache.spark.deploy.worker.Worker --webui-port 8081 master.example.com:7077 Record the Spark version, Java version, deployment mode (standalone, YARN, Kubernetes), and the master URL. Use hostname -I to get all IP addresses, and ip route to check the default gateway. Document the cluster topology: which nodes are drivers, executors, and external services (like HDFS, Kafka, or databases). If any command fails or output is unexpected, note it. For example, if spark-submit --version fails with bash: spark-submit: command not found , the Spark binaries are not in the PATH. Add them or use the full path. Checklist version and environment: - Spark version: 3.5.0 (confirmed via spark-submit --version ) - Java version: OpenJDK 11.0.20 (must be supported by Spark version) - Deployment mode: standalone cluster - Master node: master.example.com, IP 192.168.1.10, port 7077 - Worker nodes: worker1.example.com (192.168.1.11), worker2.example.com (192.168.1.12) - External services: HDFS NameNode at hdfs-namenode.example.com:8020, Kafka broker at kafka-broker.example.com:9092 ## Safe Configuration Path Network-related Spark configurations must be changed carefully. Start by viewing the current configuration. For a standalone cluster, check the spark-defaults.conf and spark-env.sh files on all nodes. # Show network-related settings from Spark defaults (read-only) grep -E "spark\.(driver|executor|blockManager|shuffle|network|rpc|broadcast)" /opt/spark/conf/spark-defaults.conf # Example output: # spark.driver.host 192.168.1.10 # spark.driver.port 7078 # spark.blockManager.port 7079 # spark.shuffle.service.port 7337 # spark.ui.port 4040 If you need to change a setting, follow these steps: - Identify the exact property and its current value. - Determine the required value based on your network topology. - Make a backup copy of the configuration file. - Edit the file on the relevant nodes (driver, executor, or both). - Restart only the affected component, not the whole cluster if possible. - Verify the change with appropriate commands. Example: Suppose Spark executors cannot bind to the default random port for block manager because a firewall blocks a range. You need to set a specific port. In spark-defaults.conf on executor nodes, add: spark.blockManager.port 40000 Before making the change, check whether the port is free: # Check if port 40000 is listening ss -tuln | grep 40000 # Expected output: no output (port is free) After editing, restart the executor process. Then verify the executor has bound to the new port: # On the executor node, find the worker process and check its open ports ps -ef | grep spark | grep Worker # Note the PID, e.g., 5678 ss -tulnp | grep 5678 # Expected output should include a line like: # tcp LISTEN 0 128 0.0.0.0:40000 0.0.0.0:* users:(("java",pid=5678,fd=123)) If the port is still not listening, check the Spark logs for errors. Checklist safe configuration path: - Current value of spark.blockManager.port: (not set, random) - New value: 40000 - Backup file: /opt/spark/conf/spark-defaults.conf.bak.20250301 - Verification command: ss -tulnp | grep 5678 - Rollback: restore backup and restart executor ## Verification and Diagnostics Once a network issue is suspected, use a systematic verification process. The following commands help diagnose common problems. ### 1. Check listening ports # List all listening TCP ports on the node ss -tuln # Example output (excerpt): # Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process # tcp LISTEN 0 128 0.0.0.0:7077 0.0.0.0:* # tcp LISTEN 0 128 0.0.0.0:8080 0.0.0.0:* # tcp LISTEN 0 128 0.0.0.0:4040 0.0.0.0:* Ensure that Spark master is listening on 7077 (or configured port), web UI on 8080, and driver UI on 4040 when an application is running. ### 2. Test connectivity using nc or telnet From a worker node, test connectivity to the master: # Using netcat (nc) nc -zv master.example.com 7077 # Expected output: # Connection to master.example.com (192.168.1.10) 7077 port [tcp/*] succeeded! If nc is not available, use telnet : telnet master.example.com 7077 # If connected, you'll see something like: # Trying 192.168.1.10... # Connected to master.example.com. # Escape character is '^]'. # Press Ctrl+] then type quit to exit. If the connection fails, check firewall rules with iptables -L -n (or firewall-cmd --list-all ). ### 3. Verify DNS resolution # Resolve hostnames nslookup master.example.com # Expected output: # Server: 192.168.1.1 # Address: 192.168.1.1#53 # # Name: master.example.com # Address: 192.168.1.10 # Reverse lookup nslookup 192.168.1.10 # Expected output: # 10.1.168.192.in-addr.arpa name = master.example.com. Both forward and reverse DNS should be consistent. Spark often uses hostnames for communication, and mismatches can cause failures like UnknownHostException . ### 4. Check Spark application logs For a running application, view the driver logs to find connection errors: # Find driver logs (standalone mode) ls /opt/spark/work/app-*/driver-*.log # Or use the web UI to view stdout/stderr. Typical network errors include ConnectException: Connection refused , BindException: Address already in use , and UnknownHostException . Checklist verification: - Ports on master: 7077, 8080 open (confirmed) - Connectivity from worker to master: SUCCESS - DNS forward and reverse: MATCH - Driver logs: no network exceptions ## Failure Modes and Recovery Here are common Spark networking failure modes, their symptoms, and recovery steps. ### Failure 1: Executor cannot connect to master Symptom: In worker logs, you see repeated errors like: WARN Worker: Failed to connect to master master.example.com:7077 Possible causes: - Master is not running or crashed. - Network partition or firewall blocking port 7077. - Wrong master URL in worker configuration. Recovery steps: - Check if master process is running: ps -ef | grep Master - Check if port is listening: ss -tuln | grep 7077 - From worker, test connectivity: nc -zv master.example.com 7077 - If master is down, restart it: /opt/spark/sbin/start-master.sh - If firewall, allow port 7077: firewall-cmd --add-port=7077/tcp --permanent && firewall-cmd --reload - Verify worker reconnects by checking logs for Successfully registered with master ### Failure 2: Driver cannot bind to port Symptom: Application fails with java.net.BindException: Address already in use or similar. Often the driver port (default random) conflicts. Recovery: - Set explicit driver port in spark-defaults.conf or in spark-submit with --conf spark.driver.port=4041 (choose an unused port). - Check for existing processes using the port: lsof -i :4041 - If needed, kill the conflicting process or choose another port. ### Failure 3: DNS resolution failures Symptom: Error java.net.UnknownHostException: master.example.com when launching executors or accessing external services. Recovery: - Verify DNS settings on all nodes ( cat /etc/resolv.conf ). - Ensure /etc/hosts has correct entries if not using central DNS. - Test with nslookup master.example.com . - If needed, add entries to /etc/hosts temporarily and verify resolution. ### Failure 4: Network timeout during shuffle Symptom: Job fails with TimeoutException or IOException: Connection reset during shuffle. Possible causes: - Insufficient network bandwidth. - High latency between nodes. - Firewall dropping long-lived connections. Recovery: - Increase timeout settings: spark.network.timeout=300s (default 120s). - Check network performance with iperf between nodes. - Ensure firewall is not killing idle connections. Checklist failure recovery: - Identify failure mode: Executor cannot connect to master (Failure 1) - Root cause: Firewall blocking port 7077 (confirmed via nc test) - Fix applied: Added firewall rule - Verification: Worker logs show successful registration ## Operations Checklist Use this checklist as a quick reference for Spark networking troubleshooting.
StepCommand/ActionExpected ResultIf Failed
1. Check Spark versionspark-submit --versionShows version 3.5.0Fix PATH or installation
2. Check Java versionjava -versionOpenJDK 11 or supportedInstall correct Java
3. Verify master processps -ef | grep MasterMaster process runningStart master: /opt/spark/sbin/start-master.sh
4. Check listening portsss -tuln | grep -E '7077|8080'Ports listedInvestigate process or configuration
5. Test connectivity from workernc -zv master.example.com 7077Connection succeededCheck network and firewall
6. Test DNS resolutionnslookup master.example.comReturns IP 192.168.1.10Fix DNS or /etc/hosts
7. Check Spark configurationgrep spark.driver.host /opt/spark/conf/spark-defaults.confCorrect hostname or IPAdjust configuration and restart
8. Review driver logstail -n 50 /opt/spark/work/app-/driver-.logNo network exceptionsDiagnose specific error
9. Check firewall rulesfirewall-cmd --list-allPorts 7077, 8080 openAdd rules
10. Verify application runspark-submit --class org.apache.spark.examples.SparkPi --master spark://master.example.com:7077 /opt/spark/examples/jars/spark-examples_2.12-3.5.0.jar 10Job completes with outputInvestigate errors in logs
Using the checklist: - Perform steps 1-4 on the master node first. - Perform steps 5-6 from a worker node. - Check configuration and logs on all relevant nodes. - After making changes, rerun the failed step to confirm. ## Conclusion Apache Spark networking troubleshooting requires a methodical approach. Start by gathering version and environment details, then check connectivity, DNS, ports, and configuration. Use the provided commands and checklists to identify and resolve common issues like connection failures, bind exceptions, and DNS problems. Always separate observation from intervention. Capture current state with read-only commands before making changes. Limit changes to one scoped item at a time, and verify each change. Document your recovery path so that you can roll back if needed. For further reading, consult the official Apache Spark documentation for your version, especially the configuration and deployment sections. Also review network-related settings for your resource manager (YARN, Kubernetes) and any external services like HDFS or Kafka that Spark interacts with. By following this guide, you can quickly diagnose and fix Spark networking issues, keeping your data pipelines running smoothly.