HDFS Production Operations Checklist with Practical Examples

Intro

Running Hadoop Distributed File System (HDFS) in production demands disciplined operational practices. A single misconfiguration or missed maintenance task can lead to data loss, downtime, or performance degradation. This article provides a practical checklist for HDFS production operations, distilled from field experience and best practices. It covers environment inventory, safe configuration, verification, failure recovery, and routine operations, with concrete commands and expected outputs. Use this checklist to reduce risk, streamline maintenance, and ensure your HDFS deployment remains healthy and reliable.

HDFS is the storage backbone for many big data platforms, including Apache Spark, NiFi, and Kafka (via Kafka Connect HDFS sink). Its architecture, with a NameNode managing metadata and DataNodes storing blocks, requires careful attention to configuration, monitoring, and maintenance.

This guide is for developers, DevOps consultants, and technical startup teams who manage HDFS clusters. It assumes a basic understanding of HDFS concepts but focuses on production-ready practices. Each section provides actionable steps, commands, and verification methods. We also highlight common pitfalls and how to avoid them.

Version and Environment Inventory

Before making any changes, you must know exactly what you are running. An accurate inventory is the foundation of safe operations.

Check Hadoop and HDFS Version

Use the hdfs version command to display the version and build information.

hdfs version

Expected output (example for Hadoop 3.3.6):

Hadoop 3.3.6
Source code repository https://github.com/apache/hadoop.git -r 1be78238728da9266a4f88195058f08fd012bf9c
Compiled by ubuntu on 2023-06-18T15:24Z
Compiled with protoc 3.7.1
From source with checksum 5d8d8b9f3b46f3a35e0b2c90b1c50b8
This command was run using /opt/hadoop/share/hadoop/common/hadoop-common-3.3.6.jar

Note the version because upgrade paths and compatibility depend on it.

Document Cluster Topology

Record the following:

• Number of NameNodes (active and standby if high availability is enabled)
• Number of DataNodes
• Rack awareness configuration
• Storage capacity per DataNode
• Network layout

You can get a quick overview from the NameNode web UI (typically port 9870) or via the command:

hdfs dfsadmin -report

This report shows live and dead nodes, capacity, and block information. Keep a copy for change tracking.

Expected output excerpt:

Live datanodes (3):

Name: 192.168.1.101:9866 (datanode1)
Hostname: datanode1
Decommission Status : Normal
Configured Capacity: 107374182400 (100 GB)
DFS Used: 21474836480 (20 GB)
Non DFS Used: 10737418240 (10 GB)
DFS Remaining: 75161927680 (70 GB)
...

Identify Dependencies and Clients

List applications that read from or write to HDFS. This includes Spark jobs, NiFi flows, Kafka connectors, and custom applications. Check their compatibility with your HDFS version, especially when planning upgrades.

Example dependency matrix:

Prerequisites Checklist

Ensure the following are in place:

• Java version compatible with Hadoop (e.g., Java 8 or 11, depending on Hadoop version)
• Adequate disk space for logs and metadata
• Time synchronization (NTP) across all nodes (critical for HDFS operations)
• Proper user and group permissions for HDFS directories
• Backup and restore procedures tested

Owner: Cluster administrator (e.g., Priya Shah, Engineering Lead). Review this inventory before any major change and update it quarterly.

Safe Configuration Path

Configuration changes should be incremental, tested, and reversible. Avoid big-bang modifications.

Understand Configuration Files

HDFS configuration is primarily in hdfs-site.xml and core-site.xml. Key parameters include:

• dfs.namenode.name.dir: directories for NameNode metadata
• dfs.datanode.data.dir: directories for DataNode blocks
• dfs.replication: default replication factor
• dfs.blocksize: block size in bytes
• dfs.namenode.handler.count: RPC handler threads

Use a Configuration Management System

Maintain configuration files in version control (e.g., Git). Use tools like Ansible, Puppet, or Chef to deploy changes consistently.

Example Git workflow:

git clone [email protected]:myorg/hadoop-config.git
cd hadoop-config
# edit hdfs-site.xml
git commit -am "Increase replication factor to 3"
git push
# then run Ansible playbook to deploy

Make Changes in a Staging Environment First

Before applying to production, test changes in a staging cluster that mirrors production. This catches issues early.

Example: Increasing Replication Factor Safely

Suppose you want to increase the default replication factor from 2 to 3 for better durability.

1. In hdfs-site.xml, change:

<property>
  <name>dfs.replication</name>
  <value>3</value>
</property>

1. Restart HDFS (or use rolling restart for HA). Verify the new setting:

hdfs getconf -confKey dfs.replication

Expected output:

3

1. For existing files, the new replication factor does not automatically apply. You must set replication explicitly or run a balancer/re-replication job. For a specific file:

hdfs dfs -setrep -w 3 /path/to/file

The -w flag waits for replication to complete.

1. Verify with:

hdfs dfs -stat %r /path/to/file

Expected output if replication completed:

3

Rolling Configuration Changes

For high-availability clusters, use rolling restarts to avoid downtime. For example, to restart DataNodes one by one:

hdfs dfsadmin -rollingUpgrade prepare
# then restart DataNodes gradually

Consult the Hadoop documentation for the exact procedure.

Configuration Pitfalls to Avoid

• Modifying NameNode directories while running: Never change dfs.namenode.name.dir without proper backup and offline procedures.
• Insufficient handler threads: Too few handlers can cause RPC congestion. Increase dfs.namenode.handler.count gradually and monitor.
• Ignoring memory settings: NameNode heap size must accommodate metadata. Use HADOOP_NAMENODE_OPTS to set appropriate -Xmx.

Owner: Configuration changes must be approved by the cluster owner (e.g., Priya Shah) and reviewed weekly in the operations meeting.

Verification and Diagnostics

After any change, verify cluster health. Use multiple tools to cross-check.

Check Cluster Summary

Use hdfs dfsadmin -report to get a high-level view.

Example excerpt:

Live datanodes (3):

Name: 192.168.1.101:9866 (datanode1)
Hostname: datanode1
Decommission Status : Normal
Configured Capacity: 107374182400 (100 GB)
DFS Used: 21474836480 (20 GB)
Non DFS Used: 10737418240 (10 GB)
DFS Remaining: 75161927680 (70 GB)

Check for any dead nodes or decommissioning nodes.

Verify NameNode Health

Access the NameNode web UI (http://namenode:9870) and check:

• Safe mode status (should be off in normal operation)
• Number of live DataNodes
• Capacity and blocks
• Any alerts in the log

Alternatively, use JMX or the hdfs haadmin command for HA clusters:

hdfs haadmin -getServiceState nn1

Expected output:

active

Check DataNode Health

On each DataNode, verify the process is running and logs are clean.

Use jps to list Java processes:

jps

Expected output includes DataNode and NodeManager (if YARN is used).

File System Checks

Run fsck to detect missing or corrupt blocks:

hdfs fsck / -files -blocks -locations

The output shows the health of the file system. Look for "Health is GOOD". For detailed block report:

hdfs fsck / -blockId blk_1073741825

Performance Diagnostics

Monitor key metrics:

• NameNode RPC queue length
• DataNode disk I/O
• Network throughput
• JVM garbage collection pauses

Use tools like Prometheus with Hadoop metrics sink, or Cloudera Manager/Ambari if available.

Example metrics to track with Prometheus:

Log Analysis

Check logs for errors:

tail -f /var/log/hadoop-hdfs/hdfs-audit.log

Look for ERROR or FATAL in hadoop-hdfs-namenode-*.log.

Owner: Verification tasks are carried out by the on-call engineer (e.g., Alex Chen) and reviewed daily.

Failure Modes and Recovery

Even with careful planning, failures occur. Know how to recover.

Common Failure Modes

NameNode Metadata Corruption

If the NameNode fails to start due to metadata corruption, you may need to restore from a checkpoint or backup. For HA clusters, fail over to standby.

Recovery steps (non-HA):

1. Stop NameNode.
2. Restore fsimage and edits from backup (e.g., from a secondary NameNode checkpoint).
3. Restart NameNode.

Example restoration command:

# Assuming backup is stored in /backup/namenode
cp /backup/namenode/current/fsimage_* /data/namenode/current/
cp /backup/namenode/current/edits_* /data/namenode/current/
# restart NameNode

DataNode Disk Failure

A disk failure on a DataNode can cause block corruption. HDFS replicates blocks, so data may still be available. Replace the disk and re-add the DataNode.

Steps:

1. Decommission the DataNode:

hdfs dfsadmin -refreshNodes

1. Wait for decommissioning to complete (check dfsadmin -report).
2. Replace disk and restart DataNode.
3. Re-commission the node.

Network Partition

If a network partition occurs, some DataNodes may become unreachable. HDFS handles this by marking them dead after a timeout. Ensure network redundancy.

High Availability Failover Issues

In HA setups, automatic failover may not trigger. Use hdfs haadmin -failover manually if needed.

hdfs haadmin -failover nn1 nn2

Rollback Procedures

Configuration Rollback

If a configuration change causes problems, revert to the previous version in your configuration management system and restart affected services.

Example:

git revert <commit-hash>
# redeploy configuration

Upgrade Rollback

Hadoop supports rolling upgrades for HDFS. For major upgrades, you can rollback using:

hdfs dfsadmin -rollingUpgrade rollback

This reverts the file system to the previous version if a rolling upgrade is in progress. Ensure you have a backup before any upgrade.

Backup and Restore Strategies

Regular backups are crucial. Options:

• DistCp: Copy data to another cluster or cloud storage.

hadoop distcp hdfs://source/path hdfs://target/path

• Snapshots: HDFS snapshots provide point-in-time copies of directories.

Enable snapshots on a directory:

hdfs dfs -createSnapshot /data snap1

Restore from snapshot:

hdfs dfs -cp /data/.snapshot/snap1/file /data/file

• NameNode metadata backups: Use dfs.namenode.name.dir with multiple directories on different disks, or use secondary NameNode checkpoints.

Recovery Verification

After any recovery, run fsck and check cluster health to ensure integrity.

Owner: Recovery procedures are owned by the on-call engineer, but the cluster owner reviews incident reports monthly.

Operations Checklist

This is a summary checklist for routine operations. Adapt it to your environment.

Daily Checks

• [ ] Check NameNode web UI for alerts
• [ ] Run hdfs dfsadmin -report and review live/dead nodes
• [ ] Monitor disk space on DataNodes
• [ ] Review logs for errors
• [ ] Check HDFS audit logs for unauthorized access

Owner: On-call engineer (e.g., Alex Chen)

Weekly Checks

• [ ] Run hdfs fsck / to detect block issues
• [ ] Check for under-replicated blocks: hdfs fsck / | grep 'Under replicated'
• [ ] Review NameNode heap usage and GC logs
• [ ] Verify backups completed (if applicable)

Owner: Cluster administrator (e.g., Priya Shah)

Monthly Checks

• [ ] Test restore from backup in a staging environment
• [ ] Review configuration for deprecated parameters
• [ ] Check for capacity trends and plan expansion
• [ ] Verify high availability failover (if applicable)

Owner: Cluster administrator and engineering lead

Quarterly/Annual Tasks

• [ ] Perform rolling upgrade tests
• [ ] Review security policies and Kerberos tickets
• [ ] Conduct disaster recovery drill
• [ ] Update documentation and runbooks

Owner: Engineering lead and security officer

Example: Automating Checks with a Script

Create a simple script to run daily checks and email results:

#!/bin/bash
# daily_hdfs_check.sh
DATE=$(date +%Y-%m-%d)
REPORT=$(hdfs dfsadmin -report)
echo "$REPORT" | mail -s "HDFS Daily Report $DATE" [email protected]

Add this to cron:

0 7 * * * /path/to/daily_hdfs_check.sh

Key Metrics to Monitor

Use a monitoring system to alert on:

• NameNode heap usage > 80%
• DataNode disk usage > 85%
• Under-replicated blocks > 0 for extended period
• RPC processing time > 100ms average
• JournalNode sync latency (HA)

Common Pitfalls and How to Avoid Them

Even experienced administrators fall into traps. Here are the most common ones and how to steer clear.

Ignoring Capacity Planning

Why it happens: Teams focus on immediate needs and fail to project data growth.

How to avoid: Monitor capacity trends monthly. Set alerts when usage exceeds 70% to plan expansion.

Recovery: If a DataNode reaches full capacity, add new DataNodes or rebalance the cluster using hdfs balancer.

Neglecting Backup Verification

Why it happens: Backups are set up but never tested, leading to false confidence.

How to avoid: Schedule monthly restore tests in a staging environment.

Recovery: If backup data is corrupt, investigate the backup pipeline and repair it. Consider using snapshots for point-in-time recovery.

Misconfiguring Replication Factor

Why it happens: Lowering replication factor to save space without considering durability.

How to avoid: Always maintain at least replication factor 3 for critical data. Use erasure coding for less critical data to save space.

Recovery: If data is under-replicated, run hdfs fsck / -replicate to trigger replication of missing blocks.

Overlooking Security Updates

Why it happens: Security patches are not applied regularly, leaving clusters vulnerable.

How to avoid: Subscribe to Hadoop security announcements and schedule quarterly patch reviews.

Recovery: Apply security patches in a staging environment first, then roll out to production using rolling restarts.

Not Documenting Changes

Why it happens: Changes are made on the fly without updating documentation, leading to confusion.

How to avoid: Require documentation as part of the change management process. Use version control for configuration files.

Recovery: If undocumented changes cause issues, compare current configuration with version history to identify differences.

Conclusion

Operating HDFS in production requires a systematic approach. By following the checklist in this article, you can minimize risks and keep your cluster healthy. Start with a thorough environment inventory, make configuration changes safely with testing and version control, verify every change with concrete commands, and prepare for failures with documented recovery procedures.

Remember to adapt this checklist to your specific environment and update it as your cluster evolves. Regular reviews and drills will ensure your team is ready for any incident.

To implement this checklist, begin with the first narrow, measurable step: run the version and inventory commands today and document your cluster topology. Then schedule the daily checks and set up monitoring alerts. Over time, expand to weekly and monthly tasks, and conduct a disaster recovery drill. This incremental approach reduces rework and builds operational confidence.