HDFS basic commands with practical examples: from discovery to recovery

Intro

This guide turns HDFS basics into reliable, everyday operations. You will inventory your environment, run core hdfs dfs commands safely, verify results with diagnostic tools, and follow clear recovery steps for common failure modes. All examples use placeholders (no secrets) and emphasize observe-first, change-later discipline.

Scope and versions: commands apply to Apache Hadoop HDFS 2.7–3.x. Some flags vary slightly across versions; when in doubt, check hdfs help or hdfs dfs -help <subcommand> in your environment.

Version and environment inventory

Before changing anything, confirm what you are running and how it is deployed.

Prerequisites:

• Shell access to a node with Hadoop client tools
• Network access to NameNode and DataNodes
• A user principal with read access (and write/admin only when required)

Read-only discovery:

# HDFS/Hadoop client version
hdfs version
hadoop version

# Nameservices and NameNode topology (HA or single NN)
hdfs getconf -confKey dfs.nameservices
hdfs getconf -namenodes
# For HA nameservice(s), list HA NameNodes per nameservice
hdfs getconf -confKey dfs.ha.namenodes.<NAMESERVICE>

# Effective configuration for a specific key
hdfs getconf -confKey dfs.replication

# Cluster storage report (no changes)
hdfs dfsadmin -report

# Quick health of filesystem namespace and blocks (read-only)
hdfs fsck / -files -blocks -locations -racks -move -delete | head -n 50
# Note: fsck above is read-only unless -move or -delete are used; omit them for pure observation

Record timestamps and outputs. Define expected signals before changing anything. Example expectations:

• hdfs dfsadmin -report shows all expected DataNodes as Alive
• Namespace usage below your alert thresholds
• hdfs getconf -confKey dfs.replication returns the default replication factor (e.g., 3)

Core HDFS command examples

The following essentials cover most daily operations. Replace placeholders like <HDFS_PATH> and <LOCAL_PATH> with your values.

1) Discovery and listing

# List directories and files
hdfs dfs -ls /
hdfs dfs -ls -h /data/events

# Recursive list with human-readable sizes
hdfs dfs -ls -R -h /projects/<TEAM>

# File metadata and summary
hdfs dfs -stat "%n %b bytes %o owner %r repl %y mtime" /data/logs/app.log

# Disk usage (HDFS logical sizes)
hdfs dfs -du -h /warehouse/tables
hdfs dfs -dus -h /warehouse/tables  # summary only

Expected signals:

• -h shows friendly sizes (MB/GB)
• -R includes nested paths
• -stat prints file name, size, owner, replication, and modification time

2) Reading and writing data

# Upload (local -> HDFS)
hdfs dfs -mkdir -p /user/<USER>/ingest
hdfs dfs -put -f /local/path/file.csv /user/<USER>/ingest/

# Download (HDFS -> local)
hdfs dfs -get /user/<USER>/ingest/file.csv /tmp/

# Stream contents
hdfs dfs -cat /user/<USER>/ingest/file.csv | head -n 5
hdfs dfs -tail -f /logs/app/app.log  # follow last 1KB, useful for debugging

# Copy and move within HDFS
hdfs dfs -cp /data/raw/file.parquet /data/stage/file.parquet
hdfs dfs -mv /data/stage/file.parquet /data/prod/file.parquet

# Safe delete using Trash (if enabled)
hdfs dfs -rm /tmp/old.tmp
hdfs dfs -rm -r /tmp/old_dir
# Permanently delete (bypasses Trash; requires caution)
hdfs dfs -rm -r -skipTrash /tmp/old_dir

Verification:

• -ls to confirm file presence and size
• -checksum to confirm integrity if needed: hdfs dfs -checksum <HDFS_PATH>

3) Permissions and ACLs

# Ownership and mode (requires privileges)
hdfs dfs -chown <USER>:<GROUP> /projects/<TEAM>
hdfs dfs -chmod -R 750 /projects/<TEAM>

# POSIX-style listing includes permissions
hdfs dfs -ls -d /projects/<TEAM>

# ACLs (fine-grained permissions)
hdfs dfs -setfacl -m user:<ANALYST>:r-x /projects/<TEAM>/dataset
hdfs dfs -getfacl /projects/<TEAM>/dataset
# Remove an ACL entry
hdfs dfs -setfacl -x user:<ANALYST> /projects/<TEAM>/dataset

Signals:

• -ls shows mode bits; getfacl shows explicit ACL entries
• Prefer ACLs for read access without changing directory ownership

4) Replication and quotas

# Adjust replication for a file and wait for completion
hdfs dfs -setrep -w 3 /data/prod/critical.parquet

# Verify replication
hdfs fsck /data/prod/critical.parquet -files -blocks -racks | grep -i replication

# Space and namespace quotas (admin)
hdfs dfsadmin -setSpaceQuota 2t /teams/<TEAM>
hdfs dfsadmin -setQuota 1000000 /teams/<TEAM>   # 1M files+dirs

# Verify quotas and usage
hdfs dfs -count -q -h /teams/<TEAM>
# Clear quotas when no longer needed
hdfs dfsadmin -clrSpaceQuota /teams/<TEAM>
hdfs dfsadmin -clrQuota /teams/<TEAM>

Signals:

• -setrep -w blocks until replication target is met or fails
• -count -q prints quotas, consumed space, and file counts

Safe configuration path

Apply the smallest justified change, only after observing current state and defining rollback.

Supported versions: Hadoop 2.7–3.x (flags shown are common across these versions). Blast radius guidance is included per example.

Example A: Raise replication for a critical file

Goal: increase durability for one file without affecting others.

Prerequisites:

• Sufficient DataNodes and disk space to hold extra replicas
• You can write metadata changes to the path

Observation:

hdfs dfs -stat "%n repl:%r size:%b" /data/prod/critical.parquet
hdfs dfsadmin -report | grep -i "Configured Capacity\|DFS Used\|Under replicated blocks" -A2

Change (blast radius: one file):

hdfs dfs -setrep -w 4 /data/prod/critical.parquet

Verification and rollback:

# Verify
hdfs fsck /data/prod/critical.parquet -files -blocks -locations | grep -i replication
# Rollback to previous factor if needed
hdfs dfs -setrep -w 3 /data/prod/critical.parquet

Failure signals and actions:

• Stuck waiting: check hdfs dfsadmin -report for low space or dead DataNodes
• Still under-replicated: search NameNode logs and confirm network reachability

Example B: Grant read access with an ACL

Goal: allow an analyst to read a dataset without changing ownership or broadening group permissions.

Prerequisites:

• ACLs enabled (dfs.namenode.acls.enabled=true)

Observation:

hdfs dfs -ls -d /projects/<TEAM>/dataset
hdfs dfs -getfacl /projects/<TEAM>/dataset

Change (blast radius: one directory tree):

hdfs dfs -setfacl -R -m user:<ANALYST>:r-x /projects/<TEAM>/dataset

Verification and rollback:

hdfs dfs -getfacl /projects/<TEAM>/dataset | grep <ANALYST>
# Test access (as the analyst user)
hdfs dfs -ls /projects/<TEAM>/dataset
# Rollback: remove the ACL entry
hdfs dfs -setfacl -R -x user:<ANALYST> /projects/<TEAM>/dataset

Verification and diagnostics

Use read-only checks to confirm health before and after changes.

• Namespace and block health:

  hdfs fsck / -files -blocks -locations | head -n 100
  hdfs fsck /path/to/object -files -blocks -locations -racks

Signals: no missing or corrupt blocks; expected replication per file.

• Cluster capacity and node status:

  hdfs dfsadmin -report

Signals: all expected DataNodes alive; reasonable utilization; Under replicated blocks near zero.

• Safe mode state:

  hdfs dfsadmin -safemode get

Signals: should be OFF during normal write operations.

• Busy directories and heavy hitters:

  hdfs dfs -count -q -h / | sort -k3 -hr | head -n 20

Signals: quickly identify top space consumers and high inode counts.

• NameNode/JMX quick checks (if permitted):
• Web UI or JMX on http(s)://<NN_HOST>:9870 (Hadoop 3) or 50070 (Hadoop 2) for dashboards and metrics.

Failure modes and recovery

Target the minimal fix, verify, and document what changed.

1. Under-replicated or missing blocks

• Observe:

  hdfs dfsadmin -report | grep -i "Under replicated"
  hdfs fsck / -list-corruptfileblocks

• Likely causes: dead DataNodes, low space, rack imbalance
• Recover:
• Fix node health or add capacity
• Rebalance if skewed: hdfs balancer -threshold 10 (admin)
• For a specific file, temporarily reduce replication to available capacity, then restore when capacity returns
• Verify: hdfs fsck /path -files -blocks | grep -i replication

1. NameNode in safe mode (writes failing)

• Observe: hdfs dfsadmin -safemode get
• Causes: startup, low block reports, missing capacity
• Recover:
• Resolve underlying DataNode or capacity issues first
• As last resort (admin): hdfs dfsadmin -safemode leave
• Verify: upload a small file and confirm -ls shows it

1. Permission denied

• Observe: exact error message and current ACLs

  hdfs dfs -getfacl /restricted/path

• Recover:
• Use targeted ACLs: hdfs dfs -setfacl -m user:<USER>:r-x /restricted/path
• Or adjust group membership/ownership if appropriate
• Verify: user can -ls but cannot write unless granted

1. Full or near-full disk

• Observe: hdfs dfsadmin -report shows high utilization
• Recover:
• Archive or remove non-critical data: hdfs dfs -rm -r /tmp/old/* (prefer Trash)
• Increase quotas only after validating need
• Add DataNodes or expand disks where possible
• Verify: utilization drops; under-replication resolves

1. Accidental delete

• Observe: check Trash and snapshots

  hdfs dfs -ls /user/<USER>/.Trash/Current
  hdfs dfs -ls /data/project/.snapshot

• Recover:
• Restore from Trash: move file back to its original path
• If snapshots enabled (admin to allow):

    hdfs dfs -createSnapshot /data/project before_delete
    # To restore later
    hdfs dfs -restoreSnapshot /data/project before_delete

• Verify: file contents and checksums match expected values

Operations checklist

Use this repeatable sequence for any HDFS change.

• Identify version and topology: hdfs version, hdfs getconf -namenodes
• Observe health and capacity: hdfs dfsadmin -report, hdfs fsck (read-only)
• Define the smallest change and its blast radius; write rollback steps
• Execute with explicit placeholders and time-bounded commands (e.g., -setrep -w)
• Verify success using -ls, -stat, fsck, and quota checks
• Record timestamps, commands, and observed outputs
• If anything deviates, stop and diagnose before proceeding

Conclusion

Effective HDFS operations start with observation, continue with tightly scoped changes, and end with verification and a clear rollback path. Use the discovery steps to understand your version and topology, apply the core hdfs dfs commands with intention, validate with dfsadmin -report and fsck, and keep recovery options like ACL adjustments, replication tuning, Trash, and snapshots ready. Consistent execution of this workflow limits blast radius, protects sensitive values, and makes both success and failure visible.