This guide provides a practical, copy‑pasteable runbook for diagnosing and fixing common HDFS problems. It is written for developers, DevOps engineers, and platform teams that operate Hadoop‑backed data lakes or analytics clusters. Follow the workflow, run the safe commands first, and use the pilot plan to rehearse the steps on a test cluster before you need them in production.

## Workflow Overview

Use this flow during incidents:

- Quick triage to capture the state without changing it.

- Identify the failure domain: client, NameNode, DataNode, network, auth, or filesystem.

- Run targeted diagnostics for that domain.

- Apply the lowest‑risk fix first and verify.

- Escalate to deeper recovery steps only if needed.

- Record commands and outcomes so you can repeat or roll back.

## Quick Triage

Start with non‑destructive checks.

- Verify processes:

jps 

- Inspect cluster state:

 hdfs dfsadmin -report 

- Scan for block issues:

 hdfs fsck / -blocks -locations -racks -openforwrite 

- List recent operations in a path:

 hdfs dfs -ls -R /path-of-interest | tail -n 50 

- Tail logs:

 tail -F $HADOOP_LOG_DIR/hdfs/*NameNode*.log 
 tail -F $HADOOP_LOG_DIR/hdfs/*DataNode*.log 
 grep -Ei 'ERROR|FATAL|Exception' $HADOOP_LOG_DIR/hdfs/*.log 
 Capture symptoms before restarting anything. Record the output of hdfs dfsadmin -report and hdfs fsck / -blocks -locations -racks -openforwrite before any service restart.

## NameNode Issues

Symptoms:

- Clients hang on list or create.

- Web UI unavailable or shows Safe Mode.

- Exceptions like org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file ... Name node is in safe mode .

What to do, step‑by‑step:

- Check safe mode:

hdfs dfsadmin -safemode get 
 If ON because fewer than dfs.namenode.replication.min blocks are replicated, first run:

hdfs fsck / -blocks -locations -racks | grep -i 'Under replicated' 
 to see which files are affected, then fix block health (see replication section) and run:

hdfs dfsadmin -safemode leave 

- Verify NameNode process and ports:

 jps 
 netstat -plnt | grep -E '8020|50070|9870' 
 Typical NameNode RPC port 8020, HTTP UI 9870 (or 50070 on older releases). If a port is in use, stop the conflicting process or change the bind address.

- Check disk on NameNode metadata dirs (fsimage, edits):

df -h 
 Ensure free space; avoid letting metadata disks hit 100%.

Look for JournalNode connectivity errors, failed shared edits, or java.io.IOException: No space left on device .

- Review NameNode logs:

Confirm core-site.xml and hdfs-site.xml (e.g., fs.defaultFS , dfs.namenode.name.dir , dfs.namenode.edits.dir ).

- If NameNode will not start:

Never run hdfs namenode -format on a live metadata directory; it wipes the namespace.

- Do not format production metadata:

Verify each JournalNode process with:

- If using JournalNodes:

jps | grep JournalNode 
 and confirm port 8485 is listening.

- Validate after recovery:

hdfs dfsadmin -report 
 Open the NameNode web UI at http://namenode:9870 and confirm live DataNodes and healthy block count.

## DataNode & Replication

Symptoms:

- Slow reads/writes limited to certain hosts.

- UNDER_REPLICATED or MISSING blocks.

- DataNode logs show Volume failed or disk I/O errors.

Recovery workflow:

- Cluster view:

hdfs dfsadmin -report 
 Shows Live/Dead decommissioning status and capacity.

- Identify bad blocks:

hdfs fsck / -blocks -locations -racks | grep -i 'Under replicated\|Missing' 

- Trigger replication for a path:

 hdfs dfs -setrep -w 3 /data/critical 
 The -w flag blocks until the target replication factor is reached.

Verify connectivity with:

- If many nodes are dead:

nc -vz datanode-host 50010 
 and inspect DataNode logs for java.io.IOException: No space left on device .

dfs.datanode.data.dir should exist, be writable by the DataNode user, and have free space.

- Verify DataNode data dirs:

Update the exclude file ( dfs.hosts.exclude ), then run:

- Decommissioning stuck:

hdfs dfsadmin -refreshNodes 
 Ensure the cluster can still meet the replication factor before the node is fully removed.

- Rebalance after adding capacity:

hdfs balancer -threshold 10 
 Monitor with hdfs balancer -threshold 10 2>&1 | tail -f ; pause with hdfs balancer -threshold 10 -pause if job latency spikes.

- Confirm block health:

hdfs fsck / -blocks | grep -v 'Under replicated' 
 A clean output means no under‑replicated blocks remain.

## Disk & Permissions

Disk full or permission issues often cause cascading failures.

### Disk space (OS level)

- df -h on NameNode and DataNodes.

- du -sh /hadoop/dfs/* to find large directories.

- Rotate or gzip large logs. Example: free space on a full DataNode volume:

logrotate -f /etc/logrotate.d/hadoop 
 gzip /var/log/hadoop/hdfs/audit.log.* 

- DataNode logs may show No space left on device or too many failed volumes. Free space, add capacity, or adjust dfs.datanode.failed.volumes.tolerated with care.

### HDFS usage and quotas

- hdfs dfs -du -h -s /projects/teamX

- hdfs dfs -count -q -h /projects/teamX

- Set a quota on a project directory:

 hdfs dfsadmin -setQuota 1000000 /projects/teamX 
 hdfs dfsadmin -setSpaceQuota 500g /projects/teamX 

- Trash cleanup when safe: hdfs dfs -expunge

- Consider fs.trash.interval for automated trash behavior.

### Permissions

- Inspect and fix:

 hdfs dfs -ls -R /path 
 hdfs dfs -chown -R user:group /path 
 hdfs dfs -chmod -R 755 /path 

- For shared temporary directories, use the sticky bit:

 hdfs dfs -chmod 1777 /tmp 

- Ensure service users own and can write to their working directories.

## Network & RPC

 Network problems lead to timeouts, dead nodes, and flaky clients.

- DNS and reachability:

getent hosts namenode.example.com 
 ping -c 3 namenode.example.com 
 Ensure reverse DNS resolves to the same FQDN.

- Required ports:

nc -vz namenode 8020 
 nc -vz datanode 50010 
 Also check 50070 or 9870 (HTTP/HTTPS), and 9864 or 50075 (DataNode web). Open firewalls accordingly.

Look for CallTimeoutException in client logs. If the default 60 seconds is too short, increase the socket timeout in hdfs-site.xml :

- Client timeouts:

<property>
 <name>dfs.client.socket-timeout</name>
 <value>120000</value>
</property> 
 Also consider tuning ipc.client.connect.max.retries for unstable links.

fs.defaultFS in core-site.xml should match hdfs://cluster and point to the active NameNode.

- Correct NameNode address:

Confirm the topology script ( /etc/hadoop/conf/topology.sh ) is executable on every node and returns the same rack name for a given host.

- Rack awareness:

## Kerberos Auth

Auth failures present as SASL or GSS errors.

Common errors:

- GSS initiate failed

- Failed to find any Kerberos tgt

- Client cannot authenticate via SASL

Recovery steps:

- Get a valid ticket:

kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs/namenode.example.com@EXAMPLE.COM 
 Verify:

klist -e 
 Run chronyc tracking or ntpstat on each node; the System time offset should be < 5 seconds.

- Fix clock skew:

Service configs often use _HOST , e.g., hdfs/_HOST@REALM . Ensure keytabs match the host FQDN:

- Validate principals and keytabs:

klist -kt /etc/security/keytabs/hdfs.headless.keytab | grep hdfs/ 
 hadoop.security.authentication set to kerberos in core-site.xml . JAAS and keytab files readable by the service user.

- Confirm Kerberos settings:

If a single DataNode fails auth, regenerate its keytab, restart only the DataNode:

- Host‑specific issues:

systemctl restart hadoop-hdfs-datanode 
 then re‑test with:

hdfs dfs -ls / 

## Safe Commands

 Read‑only or low‑risk commands to prefer early:

### Cluster state

hdfs dfsadmin -report 
 hdfs fsck / -blocks -locations -racks -openforwrite 

### Files and quotas

 hdfs dfs -ls -R /path 
 hdfs dfs -du -h -s /path 
 hdfs dfs -count -q -h /path 

### Process and ports

 jps 
 netstat -plnt | grep -E '8020|50010|50070|9870|9864|8485' 
 ss -lptn 

### Logs

 tail -F $HADOOP_LOG_DIR/hdfs/* 
 grep -Ei 'ERROR|FATAL|SASL|Under replicated|SafeMode' $HADOOP_LOG_DIR/hdfs/*.log 

### Network

 getent hosts host 
 nslookup host 
 nc -vz host port 
 curl -I http://namenode:9870/ 

### Kerberos

 klist 
 kinit -R 
 kvno hdfs/host@REALM 

## Local Pilot Plan

 Prove this workflow safely on one host or a small test cluster.

Scope: validate triage and two recovery paths (safe mode exit and under‑replicated fix).

- Baseline health:

hdfs dfsadmin -report 
 hdfs fsck / -blocks 
 In a test setup, cause safe mode by temporarily stopping a test DataNode:

- Safe mode drill:

systemctl stop hadoop-hdfs-datanode 
 Observe hdfs dfsadmin -safemode get and relevant NameNode logs for SafeModeException . Restart the DataNode and exit safe mode:

hdfs dfsadmin -safemode leave 
 Create a 10 MB test file:

- Under‑replication drill:

dd if=/dev/zero of=testfile bs=1M count=10 
 hdfs dfs -put testfile /tmp/ 
 Set replication factor to 3 and wait:

hdfs dfs -setrep -w 3 /tmp/testfile 
 Verify with:

hdfs fsck /tmp/testfile -blocks -locations 

- Kerberos drill (if enabled):

 kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs/namenode.example.com@EXAMPLE.COM 
 klist 
 Expire the TGT ( kdestroy ) or wait for expiry; observe failures; renew and re‑test.

- Record commands and timings so the team can run the same steps consistently.

## Conclusion

Effective HDFS troubleshooting follows a predictable pattern: triage with safe reads, isolate the failing domain, apply targeted diagnostics, and use the lowest‑risk recovery that addresses root cause. Start with the pilot plan to build team muscle memory. As next steps, keep a short checklist of safe commands, track common log signatures for your environment, and practice one scenario per sprint so incidents become routine to resolve.