HDFS Troubleshooting with Practical Examples: A Runbook for Operators

This guide provides a practical, copy‑pasteable runbook for diagnosing and fixing common HDFS problems. It is written for developers, DevOps engineers, and platform teams that operate Hadoop‑backed data lakes or analytics clusters. Follow the workflow, run the safe commands first, and use the pilot plan to rehearse the steps on a test cluster before you need them in production.

Workflow Overview

Use this flow during incidents:

1. Quick triage to capture the state without changing it.
2. Identify the failure domain: client, NameNode, DataNode, network, auth, or filesystem.
3. Run targeted diagnostics for that domain.
4. Apply the lowest‑risk fix first and verify.
5. Escalate to deeper recovery steps only if needed.
6. Record commands and outcomes so you can repeat or roll back.

Quick Triage

Start with non‑destructive checks.

• Verify processes:

jps

• Inspect cluster state:

hdfs dfsadmin -report

• Scan for block issues:

hdfs fsck / -blocks -locations -racks -openforwrite

• List recent operations in a path:

hdfs dfs -ls -R /path-of-interest | tail -n 50

• Tail logs:

tail -F $HADOOP_LOG_DIR/hdfs/*NameNode*.log

tail -F $HADOOP_LOG_DIR/hdfs/*DataNode*.log

grep -Ei 'ERROR|FATAL|Exception' $HADOOP_LOG_DIR/hdfs/*.log

Capture symptoms before restarting anything. Record the output of hdfs dfsadmin -report and hdfs fsck / -blocks -locations -racks -openforwrite before any service restart.

NameNode Issues

Symptoms:

• Clients hang on list or create.
• Web UI unavailable or shows Safe Mode.
• Exceptions like org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file ... Name node is in safe mode.

What to do, step‑by‑step:

1. Check safe mode:

hdfs dfsadmin -safemode get

If ON because fewer than dfs.namenode.replication.min blocks are replicated, first run:

hdfs fsck / -blocks -locations -racks | grep -i 'Under replicated'

to see which files are affected, then fix block health (see replication section) and run:

hdfs dfsadmin -safemode leave

1. Verify NameNode process and ports:

jps

netstat -plnt | grep -E '8020|50070|9870'

Typical NameNode RPC port 8020, HTTP UI 9870 (or 50070 on older releases). If a port is in use, stop the conflicting process or change the bind address.

1. Check disk on NameNode metadata dirs (fsimage, edits):

df -h

Ensure free space; avoid letting metadata disks hit 100%.

Look for JournalNode connectivity errors, failed shared edits, or java.io.IOException: No space left on device.

1. Review NameNode logs:

Confirm core-site.xml and hdfs-site.xml (e.g., fs.defaultFS, dfs.namenode.name.dir, dfs.namenode.edits.dir).

1. If NameNode will not start:

Never run hdfs namenode -format on a live metadata directory; it wipes the namespace.

1. Do not format production metadata:

Verify each JournalNode process with:

1. If using JournalNodes:

jps | grep JournalNode

and confirm port 8485 is listening.

1. Validate after recovery:

hdfs dfsadmin -report

Open the NameNode web UI at http://namenode:9870 and confirm live DataNodes and healthy block count.

DataNode & Replication

Symptoms:

• Slow reads/writes limited to certain hosts.
• UNDER_REPLICATED or MISSING blocks.
• DataNode logs show Volume failed or disk I/O errors.

Recovery workflow:

1. Cluster view:

hdfs dfsadmin -report

Shows Live/Dead decommissioning status and capacity.

1. Identify bad blocks:

hdfs fsck / -blocks -locations -racks | grep -i 'Under replicated\|Missing'

1. Trigger replication for a path:

hdfs dfs -setrep -w 3 /data/critical

The -w flag blocks until the target replication factor is reached.

Verify connectivity with:

1. If many nodes are dead:

nc -vz datanode-host 50010

and inspect DataNode logs for java.io.IOException: No space left on device.

dfs.datanode.data.dir should exist, be writable by the DataNode user, and have free space.

1. Verify DataNode data dirs:

Update the exclude file (dfs.hosts.exclude), then run:

1. Decommissioning stuck:

hdfs dfsadmin -refreshNodes

Ensure the cluster can still meet the replication factor before the node is fully removed.

1. Rebalance after adding capacity:

hdfs balancer -threshold 10

Monitor with hdfs balancer -threshold 10 2>&1 | tail -f; pause with hdfs balancer -threshold 10 -pause if job latency spikes.

1. Confirm block health:

hdfs fsck / -blocks | grep -v 'Under replicated'

A clean output means no under‑replicated blocks remain.

Disk & Permissions

Disk full or permission issues often cause cascading failures.

Disk space (OS level)

• df -h on NameNode and DataNodes.
• du -sh /hadoop/dfs/* to find large directories.
• Rotate or gzip large logs. Example: free space on a full DataNode volume:

logrotate -f /etc/logrotate.d/hadoop

gzip /var/log/hadoop/hdfs/audit.log.*

• DataNode logs may show No space left on device or too many failed volumes. Free space, add capacity, or adjust dfs.datanode.failed.volumes.tolerated with care.

HDFS usage and quotas

• hdfs dfs -du -h -s /projects/teamX
• hdfs dfs -count -q -h /projects/teamX
• Set a quota on a project directory:

hdfs dfsadmin -setQuota 1000000 /projects/teamX

hdfs dfsadmin -setSpaceQuota 500g /projects/teamX

• Trash cleanup when safe: hdfs dfs -expunge
• Consider fs.trash.interval for automated trash behavior.

Permissions

• Inspect and fix:

hdfs dfs -ls -R /path

hdfs dfs -chown -R user:group /path

hdfs dfs -chmod -R 755 /path

• For shared temporary directories, use the sticky bit:

hdfs dfs -chmod 1777 /tmp

• Ensure service users own and can write to their working directories.

Network & RPC

Network problems lead to timeouts, dead nodes, and flaky clients.

1. DNS and reachability:

getent hosts namenode.example.com

ping -c 3 namenode.example.com

Ensure reverse DNS resolves to the same FQDN.

1. Required ports:

nc -vz namenode 8020

nc -vz datanode 50010

Also check 50070 or 9870 (HTTP/HTTPS), and 9864 or 50075 (DataNode web). Open firewalls accordingly.

Look for CallTimeoutException in client logs. If the default 60 seconds is too short, increase the socket timeout in hdfs-site.xml:

1. Client timeouts:

<property>
  <name>dfs.client.socket-timeout</name>
  <value>120000</value>
</property>

Also consider tuning ipc.client.connect.max.retries for unstable links.

fs.defaultFS in core-site.xml should match hdfs://cluster and point to the active NameNode.

1. Correct NameNode address:

Confirm the topology script (/etc/hadoop/conf/topology.sh) is executable on every node and returns the same rack name for a given host.

1. Rack awareness:

Kerberos Auth

Auth failures present as SASL or GSS errors.

Common errors:

• GSS initiate failed
• Failed to find any Kerberos tgt
• Client cannot authenticate via SASL

Recovery steps:

1. Get a valid ticket:

kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs/[email protected]

Verify:

klist -e

Run chronyc tracking or ntpstat on each node; the System time offset should be < 5 seconds.

1. Fix clock skew:

Service configs often use _HOST, e.g., hdfs/_HOST@REALM. Ensure keytabs match the host FQDN:

1. Validate principals and keytabs:

klist -kt /etc/security/keytabs/hdfs.headless.keytab | grep hdfs/

hadoop.security.authentication set to kerberos in core-site.xml. JAAS and keytab files readable by the service user.

1. Confirm Kerberos settings:

If a single DataNode fails auth, regenerate its keytab, restart only the DataNode:

1. Host‑specific issues:

systemctl restart hadoop-hdfs-datanode

then re‑test with:

hdfs dfs -ls /

Safe Commands

Read‑only or low‑risk commands to prefer early:

Cluster state

hdfs dfsadmin -report

hdfs fsck / -blocks -locations -racks -openforwrite

Files and quotas

hdfs dfs -ls -R /path

hdfs dfs -du -h -s /path

hdfs dfs -count -q -h /path

Process and ports

jps

netstat -plnt | grep -E '8020|50010|50070|9870|9864|8485'

ss -lptn

Logs

tail -F $HADOOP_LOG_DIR/hdfs/*

grep -Ei 'ERROR|FATAL|SASL|Under replicated|SafeMode' $HADOOP_LOG_DIR/hdfs/*.log

Network

getent hosts host

nslookup host

nc -vz host port

curl -I http://namenode:9870/

Kerberos

klist

kinit -R

kvno hdfs/host@REALM

Local Pilot Plan

Prove this workflow safely on one host or a small test cluster.

Scope: validate triage and two recovery paths (safe mode exit and under‑replicated fix).

1. Baseline health:

hdfs dfsadmin -report

hdfs fsck / -blocks

In a test setup, cause safe mode by temporarily stopping a test DataNode:

1. Safe mode drill:

systemctl stop hadoop-hdfs-datanode

Observe hdfs dfsadmin -safemode get and relevant NameNode logs for SafeModeException. Restart the DataNode and exit safe mode:

hdfs dfsadmin -safemode leave

Create a 10 MB test file:

1. Under‑replication drill:

dd if=/dev/zero of=testfile bs=1M count=10

hdfs dfs -put testfile /tmp/

Set replication factor to 3 and wait:

hdfs dfs -setrep -w 3 /tmp/testfile

Verify with:

hdfs fsck /tmp/testfile -blocks -locations

1. Kerberos drill (if enabled):

kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs/[email protected]

klist

Expire the TGT (kdestroy) or wait for expiry; observe failures; renew and re‑test.

1. Record commands and timings so the team can run the same steps consistently.

Conclusion

Effective HDFS troubleshooting follows a predictable pattern: triage with safe reads, isolate the failing domain, apply targeted diagnostics, and use the lowest‑risk recovery that addresses root cause. Start with the pilot plan to build team muscle memory. As next steps, keep a short checklist of safe commands, track common log signatures for your environment, and practice one scenario per sprint so incidents become routine to resolve.