## Intro

HDFS is the backbone of many data platforms, so upgrading or migrating it must be predictable, observable, and reversible. This guide gives you a field-tested approach with concrete steps, commands, and checks. You will:

- Inventory versions, topology, and dependencies.

- Choose a safe path: rolling upgrade, restart upgrade, or migration to a new cluster.

- Run guarded implementation steps with examples.

- Validate correctness and performance.

- Handle failure modes and perform rollback or recovery.

- Reuse a practical operations checklist.

The first pilot should be narrow, measurable, and easy to inspect locally before deployment to the rest of the estate. Pick a contained directory or a non-critical tenant to practice the end-to-end flow and harden your runbooks.

## Version and Environment Inventory

Before you touch anything, capture the current and target state and confirm prerequisites.

- Current and target HDFS versions (and Hadoop distribution if applicable).

- NameNode topology: single NameNode, HA (active/standby), JournalNodes, ZKFC.

- DataNodes: count, storage capacity, average utilization.

- Security: Kerberos realm, keytabs, service principals; TLS settings.

- Dependent services: YARN, Spark, Hive, HBase, NiFi, Kafka connectors, custom ingestion. Identify anything writing to HDFS that must pause or drain during cutovers.

- Network: inter-rack bandwidth and latency; cross-cluster bandwidth for migrations.

- Monitoring: metrics endpoints, logs, alerting rules.

- Backups and snapshots: fsimage/edits location, snapshot policies.

Constructed example inventory (replace with your real values):

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Component</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Current</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Target</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Host count</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Change owner</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">NameNode (HA)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.2.1</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.3.6</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">2</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">ops-hdfs</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">JournalNode</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.2.1</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.3.6</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">ops-hdfs</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">DataNode</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.2.1</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.3.6</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">24</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">ops-hdfs</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">ZKFC/ZooKeeper</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.2.1 / 3.5</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3.3.6 / 3.8</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">ops-core</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Security</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">MIT Kerberos</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">same</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">n/a</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">sec-platform</td></tr></tbody>
</table>
</div>
Prerequisites checklist:

- Confirm target version supports your features (e.g., snapshots, encryption zones).

- Ensure free space on NameNode metadata disks (at least 2x fsimage/edits for safety) and on DataNodes for block reports and compactions.

- Verify HA health if planning rolling upgrades.

- Verify DistCp is available if migrating between clusters.

- Ensure you can pause or drain writers (Spark Streaming, NiFi, Kafka HDFS sinks) and resume after validation.

## Safe Configuration Path

Pick an approach that fits your topology, risk tolerance, and downtime budget. This table helps choose.

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Path</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Prerequisites</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Typical use</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Client impact</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Rollback</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Rolling upgrade in place</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">HA NameNodes with ZKFC/JournalNodes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Large clusters needing near-zero downtime</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Short failovers; most operations continue</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Supported until finalize</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Restart (offline) upgrade</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Single NN or no HA; small clusters</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Maintenance window acceptable</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Full HDFS downtime</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Restore old binaries and metadata backups</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Migration to new cluster (DistCp)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Target cluster capacity and network</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Hardware refresh, big version jump</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Cutover window only</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Keep source read-only or writeable for fallback</td></tr></tbody>
</table>
</div>

### Guardrails for any approach

- Back up metadata: fsimage and edits.

- Snapshot critical paths.

- Quiesce heavy writers.

- Keep old binaries and configs available for quick rollback.

- Avoid finalizing upgrades until validation passes.

### Back up metadata and prepare snapshots

Run on a NameNode host with admin privileges.

# Enter safe mode for a consistent image
hdfs dfsadmin -safemode enter

# Save a fresh fsimage to NameNode local storage
hdfs dfsadmin -saveNamespace

# Fetch the fsimage to an external backup location
BACKUP_DIR=/var/backups/hdfs/$(date +%F)
mkdir -p "$BACKUP_DIR"
hdfs dfsadmin -fetchImage "$BACKUP_DIR/fsimage"

# Exit safe mode
hdfs dfsadmin -safemode leave

# Optional: also archive name dirs (adjust paths to your dfs.namenode.name.dir)
# tar -czf "$BACKUP_DIR/name-dirs.tgz" /data/nn1 /data/nn2 
 Create read-only safety nets on business-critical directories. Snapshots are cheap and protect against accidental deletes during upgrades.

# Enable snapshots where needed (one-time per directory)
hdfs dfs -allowSnapshot /data/projects

# Create a baseline snapshot before any change
hdfs dfs -createSnapshot /data/projects pre-upgrade-$(date +%F) 

### Rolling upgrade (HA) example

 Use when you have Active/Standby NameNodes and JournalNodes.

- Prepare the rolling upgrade (creates special checkpoints to allow rollback):

hdfs dfsadmin -rollingUpgrade prepare
hdfs dfsadmin -rollingUpgrade query 

- Upgrade the Standby NameNode first:

- Stop Standby NN service.

- Install new HDFS binaries and updated configs.

- Start Standby NN and ensure it becomes Standby and catches up edits.

 Example using common scripts (adjust to your service manager):

# On Standby NN host
export HADOOP_HOME=/opt/hadoop-3.3.6
# Restart only the Standby NN process via your service tooling
# systemctl restart hadoop-hdfs-namenode

# Verify it is in Standby and healthy
hdfs haadmin -getServiceState nn2 

- Fail over and upgrade the former Active:

 # Trigger a controlled failover to the upgraded Standby
hdfs haadmin -failover nn1 nn2

# Upgrade previous Active (now Standby)
# systemctl stop hadoop-hdfs-namenode on nn1
# Install new binaries/configs on nn1
# systemctl start hadoop-hdfs-namenode on nn1 

- Rolling restart DataNodes in small batches:

 # On each DataNode host, drain and restart
# systemctl restart hadoop-hdfs-datanode

# After each batch, check cluster health
hdfs dfsadmin -report | grep -E "Live datanodes|Dead datanodes"
hdfs fsck / -blocks -locations -racks | tail -n 10 

- Finalize only after complete validation (see Verification section):

 hdfs dfsadmin -rollingUpgrade finalize 
 If validation fails before finalization, you can roll back to the previous version by restarting with the old binaries; the prepared checkpoint makes it safe.

### Restart (offline) upgrade example

Use when you have a single NameNode or can accept downtime.

- Stop writers and dependent services: pause NiFi HDFS processors, stop Spark jobs, pause Kafka HDFS sinks.

- Stop HDFS cleanly (order matters):

# Stop clients (YARN services that write to HDFS, if any)
# Stop HBase/Hive Metastore if tightly coupled

# Stop DataNodes first
$HADOOP_HOME/sbin/hadoop-daemons.sh stop datanode

# Stop NameNode(s), ZKFC, JournalNodes
$HADOOP_HOME/sbin/hadoop-daemon.sh stop namenode
$HADOOP_HOME/sbin/hadoop-daemons.sh stop zkfc
$HADOOP_HOME/sbin/hadoop-daemons.sh stop journalnode 

- Back up NameNode metadata as shown earlier (fsimage, edits, and name dirs).

- Install new HDFS binaries and updated configs on all nodes.

- Start JournalNodes, ZKFC, NameNode, then DataNodes:

 $HADOOP_HOME/sbin/hadoop-daemons.sh start journalnode
$HADOOP_HOME/sbin/hadoop-daemons.sh start zkfc
$HADOOP_HOME/sbin/hadoop-daemon.sh start namenode
$HADOOP_HOME/sbin/hadoop-daemons.sh start datanode 

- Validate thoroughly. Do not delete old metadata or binaries until you finalize your decision to stay on the new version.

### Migration to a new cluster (DistCp) example

 Use when you need new hardware, a big version jump, or a clean slate. This reduces risk because the source cluster remains untouched until cutover.

Assumptions:

- Source: hdfs://src-nn:8020

- Target: hdfs://dst-nn:8020

- Kerberos enabled both sides, with cross-realm trust or matching principals.

- Prepare target directories and permissions:

# Create top-level directory and set ownership on target
hdfs dfs -fs hdfs://dst-nn:8020 -mkdir -p /data/projects
hdfs dfs -fs hdfs://dst-nn:8020 -chown -R analytics:analytics /data/projects 

- Take a baseline snapshot and run initial DistCp:

 # On source cluster
hdfs dfs -allowSnapshot /data/projects || true
hdfs dfs -createSnapshot /data/projects snap0

# Baseline copy with preservation and throttling (constructed example bandwidth)
hadoop distcp \
 -prbugp -m 50 -bandwidth 100 \
 hdfs://src-nn:8020/data/projects \
 hdfs://dst-nn:8020/data/projects 
 Options used:

- -p rbugp preserves replication, block size, user, group, permissions.

- -m 50 sets 50 map tasks (tune for your cluster).

- -bandwidth 100 limits per-map bandwidth to 100 MB/s (constructed example).

- Incremental sync during cutover window:

Create a new snapshot and use snapshot diff or update/delete pass.

# Create new snapshot on source
hdfs dfs -createSnapshot /data/projects snap1

# If your DistCp supports snapshot diff copying
hadoop distcp -prbugp -m 25 \
 -diff snap0 snap1 \
 hdfs://src-nn:8020/data/projects \
 hdfs://dst-nn:8020/data/projects

# If snapshot diff is not available, use update+delete
hadoop distcp -prbugp -m 25 -update -delete \
 hdfs://src-nn:8020/data/projects \
 hdfs://dst-nn:8020/data/projects 

- Verification and cutover:

- Run path-by-path checksums or record counts where feasible.

- Point clients to the new cluster (core-site.xml fs.defaultFS, service configs).

- Keep source in read-only mode for a defined fallback period.

## Verification and Diagnostics

 You need clear signals that the upgrade or migration is correct and healthy. Use these checks before finalizing.

### Cluster health and storage

# DataNode liveness
hdfs dfsadmin -report | egrep "Live datanodes|Dead datanodes|Under replicated blocks"

# Filesystem health (summarized)
hdfs fsck / -blocks -locations -racks | tail -n 20

# NameNode state (HA)
hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2 
 Expected results (constructed examples):

- Live datanodes equals expected host count; Dead datanodes is 0.

- Under replicated blocks is 0 or steadily decreasing to 0.

- Active/Standby roles correctly assigned; no automatic failovers occurring repeatedly.

### Rolling upgrade progress

hdfs dfsadmin -rollingUpgrade query 
 Expected: returns the prepared state during upgrade; after finalize, reports no ongoing rolling upgrade.

### Metadata integrity

# Confirm snapshots exist
hdfs lsSnapshottableDir

# Validate encryption zones (if used)
hdfs crypto -listZones 

### DistCp verification

 # Spot-verify critical directories (sizes and counts)
hdfs dfs -du -s -h hdfs://src-nn:8020/data/projects/teamA
hdfs dfs -du -s -h hdfs://dst-nn:8020/data/projects/teamA

# Optional: simple file-level checksum sampling
for f in $(hdfs dfs -ls -R hdfs://src-nn:8020/data/projects/teamA | awk '{print $8}' | head -100); do
 hdfs dfs -checksum "$f\);
 hdfs dfs -checksum "hdfs://dst-nn:8020${f#hdfs://src-nn:8020}\" ;
done 

### Client smoke tests

- Spark: read 10 representative files and count rows; write a small parquet and verify read-back.

- NiFi: run a single flow file through a sandbox path and confirm permissions.

- Kafka connectors: produce a small batch to HDFS sink and verify file appears with expected replication.

### Logs and metrics

- NameNode and DataNode logs: watch for FATAL/ERROR entries during start and block reports.

- Metrics: RPC queue time, GC pauses, edit log tail latencies, under-replication rate, NN heap usage.

## Failure Modes and Recovery

 Expect and plan for these. Where possible, detect them early and roll forward safely; otherwise, roll back cleanly.

### Common issues

- NameNode will not start after upgrade

- Symptom: startup aborts with layout version mismatch errors.

- Fix: ensure you installed matching versions across all NameNodes and JournalNodes. If offline upgrade, restore old binaries and name dirs, then reattempt with correct versions.

- DataNodes stuck upgrading or repeatedly re-registering

- Symptom: DataNodes report version/namespace ID mismatch.

- Fix: verify clusterID consistency in /dfs/dn/current/VERSION ; do not delete DataNode data unless you intend to full reformat. Restart with matching binaries.

- Under-replicated blocks after DataNode restarts

- Symptom: under-replication count spikes and does not recover.

- Fix: allow time for replication to settle. If persistent, check network partitions or disk failures; use balancer if skewed.

- Kerberos failures

- Symptom: GSSException or expired ticket errors.

- Fix: refresh keytabs and validate principal names. Run klist -k on services and kinit -kt tests. Ensure clock skew < 5 minutes.

- DistCp stalls or fails

- Symptom: mappers hang, transfer rate drops to near zero.

- Fix: reduce -m parallelism, increase -bandwidth caps, or segment directories. Check network throttling and NameNode RPC queue times.

### Rollback strategies

- Rolling upgrade : if not finalized, restart NameNodes and DataNodes with old binaries. The prepared checkpoint allows automatic downgrade. Verify with hdfs dfsadmin -rollingUpgrade query before attempting rollback.

- Restart upgrade (offline) : stop HDFS, restore previous binaries/configs, and restore fsimage/edits from backup if metadata was modified and you cannot start cleanly.

- Migration : point clients back to source cluster. If target received writes, reconcile with a reverse DistCp or treat target as disposable and re-run baseline later.

### Recovery checks after rollback

- hdfs dfsadmin -report shows expected Live datanodes and 0 Dead.

- hdfs fsck / shows no missing blocks.

- Application smoke tests succeed (reads and writes).

### When to roll forward vs roll back

- Roll forward if the cluster is healthy and only transient replication or client cache errors appear.

- Roll back if NameNode cannot start cleanly, metadata is inconsistent, or critical apps cannot read existing data.

## Operations Checklist

Use this for both upgrades and migrations. Adapt to your environment.

- Plan and inventory

- Record current and target versions and topology.

- Identify dependent services and a communication plan.

- Define a pilot scope with measurable outcomes.

- Prepare

- Back up fsimage and edits; archive name dirs.

- Create snapshots on critical directories.

- Validate Kerberos and TLS; refresh keytabs as needed.

- Quiesce or drain heavy writers (Spark, NiFi, Kafka sinks).

- Execute (choose one path)

- Rolling upgrade: prepare, upgrade Standby, fail over, upgrade former Active, roll DataNodes, query progress.

- Restart upgrade: stop services in order, upgrade binaries/configs, start in order.

- Migration: baseline DistCp, incremental sync, cutover.

- Verify

- hdfs dfsadmin -report and hdfs fsck / show healthy state.

- HA roles correct; manual failover works.

- DistCp directory sizes match; checksum spot checks pass.

- Client smoke tests succeed end to end.

- Finalize

- For rolling: finalize upgrade only after all checks pass.

- For restart: mark change complete, retain backups for a defined period.

- For migration: switch clients, set source read-only for fallback window.

- Post-ops

- Re-enable snapshots schedules and compaction jobs.

- Close the change with metrics: time to recover, any under-replication counts, and client error rates.

- Update runbooks with lessons learned.

## Practical examples and snippets

Below are concise, copy-ready snippets you can adapt.

### Verify HA and failover

hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2
hdfs haadmin -failover nn1 nn2
hdfs haadmin -getServiceState nn1
hdfs haadmin -getServiceState nn2 
 Expected: roles swap once without errors; clients continue without significant failures.

### Balance the cluster after DataNode restarts

# Run balancer at moderate threshold (constructed example 10%)
hdfs balancer -threshold 10 
 Let it run until movement stabilizes; monitor network and disk IO.

### Update client configs for cutover

<!-- In core-site.xml -->
<property>
 <name>fs.defaultFS</name>
 <value>hdfs://dst-nn:8020</value>
</property> 
 Distribute config update and restart client services.

## Conclusion

Upgrading or migrating HDFS safely is about preparation, guarded execution, and visible outcomes. Start with a small pilot you can measure, then apply the same pattern cluster-wide: inventory, back up, snapshot, execute with a reversible plan, verify with explicit checks, and only then finalize. Whether you choose a rolling in-place upgrade, a brief offline restart, or a full migration with DistCp, the examples and checklists here give you a repeatable path with clear recovery options.

Your next steps:

- Fill out your environment inventory and pick a narrow pilot directory.

- Stage metadata backups and snapshots today.

- Dry-run the verification commands in a non-critical area.

- Schedule a pilot window and rehearse rollback steps before touching production.