HDFS troubleshooting with practical examples: practical implementation guide

This guide shows how to troubleshoot HDFS with practical, copy-pasteable steps. You will learn to read the right logs, run safe commands, isolate causes, and execute step-by-step recovery workflows for common failures. Keep this as a runbook for developers, DevOps consultants, and startup teams running data lakes or Hadoop-backed services.

Workflow Overview

Use this flow during incidents:

1. Quick triage to capture the state without changing it.
2. Identify the failure domain: client, NameNode, DataNode, network, auth, or filesystem.
3. Run targeted diagnostics for that domain.
4. Apply the lowest-risk fix first and verify.
5. Escalate to deeper recovery steps only if needed.
6. Record commands and outcomes so you can repeat or roll back.

Quick Triage

Start with non-destructive checks.

• Verify processes:
• On NameNode and DataNodes: jps
• Inspect cluster state:
• hdfs dfsadmin -report
• Scan for block issues:
• hdfs fsck / -blocks -locations -racks -openforwrite
• List recent operations in a path:
• hdfs dfs -ls -R /path-of-interest | tail -n 50
• Tail logs:
• tail -F $HADOOP_LOG_DIR/hdfs/NameNode.log
• tail -F $HADOOP_LOG_DIR/hdfs/DataNode.log
• grep -Ei 'ERROR|FATAL|Exception' $HADOOP_LOG_DIR/hdfs/*.log

Capture symptoms before restarting anything.

NameNode Issues

Symptoms:

• Clients hang on list or create.
• Web UI unavailable or shows Safe Mode.
• Exceptions like:
• org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot create file ... Name node is in safe mode

What to do, step-by-step:

1. Check safe mode:

• hdfs dfsadmin -safemode get
• If ON due to low block replication, fix block health first (see replication section), then: hdfs dfsadmin -safemode leave.

1. Verify NameNode process and ports:

• jps should show NameNode.
• netstat -plnt | grep -E '8020|50070|9870'
• If a port is in use, stop the conflicting process or change the bind address.

1. Check disk on NameNode metadata dirs (fsimage, edits):

• df -h
• Ensure free space; avoid letting metadata disks hit 100%.

1. Review NameNode logs:

• Look for JournalNode connectivity errors, failed shared edits, or java.io.IOException: No space left on device.

1. If NameNode will not start:

• Confirm core-site.xml and hdfs-site.xml (e.g., fs.defaultFS, dfs.namenode.name.dir, dfs.namenode.edits.dir).

1. Do not format production metadata:

• Never run hdfs namenode -format on a live metadata directory.

1. If using JournalNodes:

• Ensure all JournalNodes are up; check port 8485 and their logs.

1. Validate after recovery:

• hdfs dfsadmin -report
• Open the NameNode web UI and confirm live DataNodes and block health.

DataNode & Replication

Symptoms:

• Slow reads/writes limited to certain hosts.
• UNDER_REPLICATED or MISSING blocks.
• DataNode logs show Volume failed or disk I/O errors.

Recovery workflow:

1. Cluster view:

• hdfs dfsadmin -report to see Live/Dead decommissioning status and capacity.

1. Identify bad blocks:

• hdfs fsck / -blocks -locations -racks | grep -i 'Under replicated\|Missing'

1. Trigger replication for a path:

• hdfs dfs -setrep -w 3 /data/critical
• The -w waits until target replication is met.

1. If many nodes are dead:

• Check network/firewall rules and DataNode logs for disk failures.

1. Verify DataNode data dirs:

• dfs.datanode.data.dir should exist, be writable by the DataNode user, and have free space.

1. Decommissioning stuck:

• Update exclude file, then hdfs dfsadmin -refreshNodes.
• Ensure target replication can be satisfied before final removal.

1. Rebalance after adding capacity:

• hdfs balancer -threshold 10
• Monitor and pause if it impacts jobs.

1. Confirm block health:

• hdfs fsck / -blocks | grep -v 'Under replicated'

Disk & Permissions

Disk full or permission issues often cause cascading failures.

Disk space (OS level):

• df -h on NameNode and DataNodes.
• du -sh /hadoop/dfs/* to find large dirs.
• Rotate or gzip large logs.
• DataNode logs may show No space left on device or too many failed volumes. Free space, add capacity, or adjust dfs.datanode.failed.volumes.tolerated with care.

HDFS usage and quotas:

• hdfs dfs -du -h -s /projects/teamX
• hdfs dfs -count -q -h /projects/teamX
• Trash cleanup when safe: hdfs dfs -expunge
• Consider fs.trash.interval for automated trash behavior.

Permissions:

• Inspect and fix:
• hdfs dfs -ls -R /path
• hdfs dfs -chown -R user: group /path
• hdfs dfs -chmod -R 755 /path
• For shared temp dirs, use sticky bit 1777 where appropriate.
• Ensure service users own and can write to their working dirs.

Network & RPC

Network problems lead to timeouts, dead nodes, and flaky clients.

1. DNS and reachability:

• getent hosts namenode.example.com
• ping -c 3 namenode.example.com
• Ensure reverse DNS resolves to the same FQDN.

1. Required ports:

• nc -vz namenode 8020
• nc -vz datanode 50010
• Also check 50070 or 9870 (HTTP/HTTPS), and 9864 or 50075 (DataNode web).
• Open firewalls accordingly.

1. Client timeouts:

• Look for CallTimeoutException in client logs.
• Consider tuning dfs.client.socket-timeout and ipc.client.connect.max.retries for unstable links.

1. Correct NameNode address:

• fs.defaultFS in core-site.xml should match hdfs://cluster and point to the active NameNode.

1. Rack awareness:

• Ensure topology scripts are present and executable across nodes, if used.

Kerberos Auth

Auth failures present as SASL or GSS errors.

Common errors:

• GSS initiate failed
• Failed to find any Kerberos tgt
• Client cannot authenticate via SASL

Recovery steps:

1. Get a valid ticket:

• kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs/host@REALM
• Verify: klist

1. Fix clock skew:

• chronyc tracking or ntpstat; keep drift small (ideally under 5 minutes).

1. Validate principals and keytabs:

• Service configs often use _HOST, e.g., hdfs/_HOST@REALM.
• Ensure keytabs match the host FQDN.

1. Confirm Kerberos settings:

• hadoop.security.authentication set to kerberos in core-site.xml.
• JAAS and keytab files readable by the service user.

1. Host-specific issues:

• Regenerate the host keytab, restart only the affected service, and re-test with hdfs dfs -ls /.

Safe Commands

Read-only or low-risk commands to prefer early:

• Cluster state:
• hdfs dfsadmin -report
• hdfs fsck / -blocks -locations -racks -openforwrite
• Files and quotas:
• hdfs dfs -ls -R /path
• hdfs dfs -du -h -s /path
• hdfs dfs -count -q -h /path
• Process and ports:
• jps
• netstat -plnt | grep -E '8020|50010|50070|9870|9864|8485'
• ss -lptn
• Logs:
• tail -F $HADOOP_LOG_DIR/hdfs/*
• grep -Ei 'ERROR|FATAL|SASL|Under replicated|SafeMode' $HADOOP_LOG_DIR/hdfs/*.log
• Network:
• getent hosts host
• nslookup host
• nc -vz host port
• curl -I http://namenode:9870/
• Kerberos:
• klist
• kinit -R
• kvno hdfs/host@REALM

Local Pilot Plan

Prove this workflow safely on one host or a small test cluster.

Scope: validate triage and two recovery paths (safe mode exit and under-replicated fix).

1. Baseline health:

• hdfs dfsadmin -report
• hdfs fsck / -blocks

1. Safe mode drill:

• In a test setup, cause safe mode by temporarily reducing replication tolerance or by stopping a test DataNode.
• Observe hdfs dfsadmin -safemode get and relevant NameNode logs.
• Bring capacity back and exit: hdfs dfsadmin -safemode leave.

1. Under-replication drill:

• Create a small file: hdfs dfs -put README /tmp/
• hdfs dfs -setrep -w 3 /tmp/README on a cluster with 3 DNs; watch replication complete.
• Verify with hdfs fsck /tmp/README -blocks -locations.

1. Kerberos drill (if enabled):

• kinit -kt ... and klist.
• Expire the TGT or wait; observe failures; renew and re-test.

1. Record commands and timings so the team can run the same steps consistently.

Conclusion

Effective HDFS troubleshooting follows a predictable pattern: triage with safe reads, isolate the failing domain, apply targeted diagnostics, and use the lowest-risk recovery that addresses root cause. Start with the pilot plan to build team muscle memory. As next steps, keep a short checklist of safe commands, track common log signatures for your environment, and practice one scenario per sprint so incidents become routine to resolve.