HDFS CI/CD Automation: A Practical Implementation Guide With Ready-to-Run Examples

Introduction

Done well, HDFS automation pays for itself quickly: fewer manual mistakes, predictable changes, and clear audit trails. This guide shows how to build practical, testable CI/CD pipelines for HDFS that you can deploy the same day. You will implement two high‑impact workflows with plan, apply, verification, and rollback:

• Manage directories, quotas, and ACLs as code.
• Roll out replication factor changes safely and verifiably.

The approach is simple: declare desired state in manifests, generate a plan, apply idempotently, and verify. Along the way you will see concrete commands, failure modes, and an operations checklist you can run on every change window.

Environment and Prerequisites

Before you automate, record the versions and endpoints so runs are consistent:

• HDFS: example 3.3.6 with HA NameNodes (nn1, nn2) and JournalNodes.
• Hadoop client CLI: example 3.3.6 available on the automation host.
• Java: example 11.x for Hadoop tools.
• Kerberos: enabled with a service principal and keytab for non‑interactive kinit.
• OS and access: a gateway host with HADOOP_CONF_DIR pointing to cluster XMLs and network access to NameNode RPC/IPC ports.
• Optional: Ranger or Sentry in place for centralized authorization.

You will need:

• A Git repository for manifests and scripts.
• A CI/CD runner that can execute shell on the gateway host.
• yq installed on the runner for YAML parsing.
• Kerberos keytab and principal environment variables if security is enabled.

A Safe Pattern for HDFS CI/CD

Adopt these principles for repeatable outcomes:

1. Declare desired state as code. Store HDFS intents (paths, quotas, ACLs, replication) in versioned manifest files, not embedded in scripts.

1. Plan first. Compute and print a diff of proposed actions without changing HDFS.

1. Apply idempotently and verify. Make operations safe to re‑run, then validate with explicit checks and have clear rollback steps.

The following examples implement this pattern end‑to‑end.

Example A: Manage directories, quotas, and ACLs as code

Goal: Ensure a set of project directories exists with defined namespace and space quotas and standard ACLs.

Manifest (example)

# file: hdfs_dirs.yaml
root: /data/projects
entries:
  - path: analytics
    ns_quota: 200000       # namespace quota (files + dirs)
    space_quota: 10t       # logical space quota
    acls:
      - user:alice:rwx
      - user:bob:r-x
      - group:analytics:r-x
  - path: ingest
    ns_quota: 100000
    space_quota: 5t
    acls:
      - user:etl:rwx
      - group:ops:r-x

Planner/apply script (bash, idempotent)

#!/usr/bin/env bash
# file: hdfs_dirs.sh
set -euo pipefail

MODE="${1:-}"   # plan | apply
MANIFEST="${2:-hdfs_dirs.yaml}"

if [[ -z "$MODE" || ! "$MODE" =~ ^(plan|apply)$ ]]; then
  echo "Usage: $0 plan|apply [manifest.yaml]" >&2
  exit 2
fi

need() { command -v "$1" >/dev/null 2>&1 || { echo "missing $1" >&2; exit 2; }; }
need hdfs
need yq

# Kerberos non-interactive login if configured
if [[ -n "${KRB5_PRINCIPAL:-}" && -n "${KRB5_KEYTAB:-}" ]]; then
  kinit -kt "$KRB5_KEYTAB" "$KRB5_PRINCIPAL" >/dev/null
fi

ts() { date +"%Y-%m-%dT%H:%M:%S%z"; }
log() { echo "[$(ts)] $*"; }

die() { echo "ERROR: $*" >&2; exit 1; }

ROOT=$(yq -r '.root' "$MANIFEST")
[[ -n "$ROOT" && "$ROOT" != "null" ]] || die "manifest missing .root"

COUNT=$(yq -r '.entries | length' "$MANIFEST")
[[ "$COUNT" =~ ^[0-9]+$ ]] || die "manifest entries invalid"

for i in $(seq 0 $((COUNT-1))); do
  SUBPATH=$(yq -r ".entries[$i].path" "$MANIFEST")
  NSQ=$(yq -r ".entries[$i].ns_quota // \"\"" "$MANIFEST")
  SPQ=$(yq -r ".entries[$i].space_quota // \"\"" "$MANIFEST")
  FULL="$ROOT/$SUBPATH"

  # Ensure directory exists
  if ! hdfs dfs -test -d "$FULL"; then
    log "CREATE dir $FULL"
    [[ "$MODE" == "apply" ]] && hdfs dfs -mkdir -p "$FULL"
  else
    log "EXISTS $FULL"
  fi

  # Read current quotas (fields: QUOTA REMAINING SPACE_QUOTA REMAINING ... PATH)
  CURRENT_LINE=$(hdfs dfs -count -q "$FULL" | tail -n1 || true)
  CUR_NSQ=$(awk '{print $1}' <<<"$CURRENT_LINE")
  CUR_SPQ=$(awk '{print $3}' <<<"$CURRENT_LINE")

  # Set namespace quota if different and desired provided
  if [[ -n "$NSQ" && "$NSQ" != "null" ]]; then
    if [[ "$CUR_NSQ" != "$NSQ" ]]; then
      log "SET ns_quota $NSQ on $FULL (was ${CUR_NSQ:-none})"
      [[ "$MODE" == "apply" ]] && hdfs dfsadmin -setQuota "$NSQ" "$FULL"
    fi
  fi

  # Set space quota if different and desired provided
  if [[ -n "$SPQ" && "$SPQ" != "null" ]]; then
    if [[ "$CUR_SPQ" != "$SPQ" ]]; then
      log "SET space_quota $SPQ on $FULL (was ${CUR_SPQ:-none})"
      [[ "$MODE" == "apply" ]] && hdfs dfsadmin -setSpaceQuota "$SPQ" "$FULL"
    fi
  fi

  # Build ACL spec from array of strings like user:alice:rwx
  ACLS_LEN=$(yq -r ".entries[$i].acls | length" "$MANIFEST" 2>/dev/null || echo 0)
  if [[ "$ACLS_LEN" =~ ^[1-9][0-9]*$ ]]; then
    ACLSPEC=$(yq -r ".entries[$i].acls[]" "$MANIFEST" | paste -sd, -)
    log "SET ACL '$ACLSPEC' on $FULL (recursive)"
    [[ "$MODE" == "apply" ]] && hdfs dfs -setfacl -R -m "$ACLSPEC" "$FULL"
  fi

done

Notes:

• The script prints what it would change in plan mode and only executes in apply mode.
• Re‑runs are safe: creating an existing directory is skipped; identical quotas or ACLs result in no change.
• For Kerberos, pass KRB5_PRINCIPAL and KRB5_KEYTAB via CI secrets. The script kinit calls only when both are present.

CI/CD skeleton (portable idea)

• Stage: lint
• Validate YAML (yamllint), lint shell (shellcheck), and fail fast.
• Stage: plan
• Run bash hdfs_dirs.sh plan hdfs_dirs.yaml and store plan.out.
• Stage: apply (manual gate)
• Run bash hdfs_dirs.sh apply hdfs_dirs.yaml and store apply.out.

Example plan output (illustrative)

[2026-08-16T10:12:05+0000] CREATE dir /data/projects/analytics
[2026-08-16T10:12:05+0000] SET ns_quota 200000 on /data/projects/analytics (was none)
[2026-08-16T10:12:05+0000] SET space_quota 10t on /data/projects/analytics (was none)
[2026-08-16T10:12:05+0000] SET ACL 'user:alice:rwx,user:bob:r-x,group:analytics:r-x' on /data/projects/analytics (recursive)
[2026-08-16T10:12:05+0000] EXISTS /data/projects/ingest
[2026-08-16T10:12:05+0000] SET ns_quota 100000 on /data/projects/ingest (was 80000)
[2026-08-16T10:12:05+0000] SET space_quota 5t on /data/projects/ingest (was 4t)
[2026-08-16T10:12:05+0000] SET ACL 'user:etl:rwx,group:ops:r-x' on /data/projects/ingest (recursive)

Rollback (Example A)

• Quotas: restore previous values with hdfs dfsadmin -setQuota <old> and -setSpaceQuota <old>.
• ACLs: remove all extended ACLs with hdfs dfs -setfacl -R -b <path> (owner and mode remain).
• Directories: remove only empty directories with hdfs dfs -rmdir <path>; never delete populated trees without a retention plan.

Example B: Safe rollout of replication factor changes

Changing replication impacts capacity and recovery SLAs. Plan first, apply in controlled windows, and verify convergence.

Manifest (example)

# file: hdfs_replication.yaml
rules:
  - path: /data/projects/analytics
    replication: 3
  - path: /data/projects/ingest
    replication: 2

Planner/apply script (bash)

#!/usr/bin/env bash
# file: hdfs_replication.sh
set -euo pipefail

MODE="${1:-}"   # plan | apply
MANIFEST="${2:-hdfs_replication.yaml}"

if [[ -z "$MODE" || ! "$MODE" =~ ^(plan|apply)$ ]]; then
  echo "Usage: $0 plan|apply [manifest.yaml]" >&2
  exit 2
fi

need() { command -v "$1" >/dev/null 2>&1 || { echo "missing $1" >&2; exit 2; }; }
need hdfs
need yq

if [[ -n "${KRB5_PRINCIPAL:-}" && -n "${KRB5_KEYTAB:-}" ]]; then
  kinit -kt "$KRB5_KEYTAB" "$KRB5_PRINCIPAL" >/dev/null
fi

ts() { date +"%Y-%m-%dT%H:%M:%S%z"; }
log() { echo "[$(ts)] $*"; }

COUNT=$(yq -r '.rules | length' "$MANIFEST")
for i in $(seq 0 $((COUNT-1))); do
  PATH_PREFIX=$(yq -r ".rules[$i].path" "$MANIFEST")
  DESIRED=$(yq -r ".rules[$i].replication" "$MANIFEST")

  log "Inspecting $PATH_PREFIX (desired $DESIRED)"
  # Sample up to 50 files for planning clarity
  SAMPLE_FILES=$(hdfs dfs -ls -R "$PATH_PREFIX" 2>/dev/null | awk '{print $8}' | head -n 50)
  CHANGES=0
  while read -r F; do
    [[ -z "$F" ]] && continue
    if hdfs dfs -test -d "$F"; then continue; fi
    CUR=$(hdfs dfs -stat %r "$F" 2>/dev/null || echo -1)
    if [[ "$CUR" -ne "$DESIRED" ]]; then
      log "DIFF $F rep=$CUR -> $DESIRED"
      CHANGES=$((CHANGES+1))
    fi
  done <<< "$SAMPLE_FILES"

  if [[ "$MODE" == "apply" ]]; then
    log "Applying replication=$DESIRED under $PATH_PREFIX (recursive, wait)"
    hdfs dfs -setrep -R -w "$DESIRED" "$PATH_PREFIX"
  fi

  log "Summary for $PATH_PREFIX: sample files needing change=$CHANGES"
done

Post‑apply verification (Example B)

• Convergence: hdfs fsck <path> -files -blocks | grep Under-replicated should return 0.
• Spot checks: hdfs dfs -stat %r <some/file> should equal the desired replication.
• Capacity watch: ensure DataNodes have headroom; consider enabling or pausing the balancer thoughtfully.

Rollback (Example B)

• Revert replication to the prior factor with hdfs dfs -setrep -R -w <old_factor> <path>.
• If capacity is tight, pause heavy jobs (DistCp, large ingests) until replication stabilizes.

Verification and Diagnostics

Run these checks before and after each apply.

Pre‑apply checks:

• NameNode and HA state:
• hdfs haadmin -getServiceState nn1 (expect one active, one standby)
• hdfs dfsadmin -safemode get (expect OFF)
• Authentication:
• klist shows a valid TGT for the CI principal
• Authorization (sanity check on one path):
• hdfs dfs -test -w /tmp returns success for the principal
• Safety net:
• Confirm fs.trash.interval in core-site.xml if you rely on Trash

Post‑apply checks:

• Example A (paths, quotas, ACLs):
• Directories exist: hdfs dfs -test -d <path>
• Quotas applied: hdfs dfs -count -q <path>; compare QUOTA and SPACE_QUOTA to the manifest
• ACLs in place: hdfs dfs -getfacl <path> | grep user:alice:rwx (adjust for your entries)
• Example B (replication):
• Under‑replicated blocks: hdfs fsck <path> -blocks -files | grep Under-replicated returns 0
• Spot files show desired %r replication

Diagnostics when something looks wrong:

• SafeMode is ON: do not apply; investigate NameNode logs and DataNode block reports.
• Permission denied for ACL or quota: check Ranger/Sentry policy; use a service principal with admin rights on metadata ops.
• Kerberos failures: kinit -kt again; ensure time sync and correct principal/realm.
• ACL recursion is slow: apply in smaller path batches or off‑peak windows.
• Replication stuck: examine DataNode health, rack awareness, and disk space; temporarily pause heavy storage jobs.

Start Small: A Short, Inspectable Pilot

Begin with a single project directory and a narrow manifest. Run plan during a low‑traffic window, review the diff, then apply. Keep your first few runs small enough that you can read the entire plan.out and apply.out in minutes. Once stable, scale to additional paths and teams.

Operations Checklist

Before you start:

• Confirm change window and on‑call coverage.
• Verify HA: one active, one standby; safemode OFF.
• Validate authentication: klist shows valid TGT for the CI principal.
• Confirm HADOOP_CONF_DIR points to the intended cluster.

Plan phase:

• Update manifests, commit, and tag if you track releases.
• Run lint; fix YAML or shell issues before proceeding.
• Run plan; review plan.out for scope and correctness.
• If changes are broad, split into smaller batches or stagger over windows.

Apply phase:

• Optionally pause heavy HDFS operations (large ingest, DistCp, balancer) to reduce noise.
• Run apply and stream logs; if repeated errors appear, stop, diagnose, and re‑plan.

Post‑apply verification:

• Re‑run key checks: quotas via dfs -count -q, ACLs via getfacl, replication via fsck and %r.
• Ensure under‑replicated blocks drop to baseline on modified paths.
• Archive apply.out, verification logs, and timings for audit and rollback.

If something fails:

• Use apply.out to identify exactly what changed.
• Revert selectively: reset quotas, remove ACLs, or restore replication factors.
• Document the issue and lessons learned; schedule a follow‑up change if needed.

Failure Modes and Safe Responses

• NameNode in SafeMode: abort apply; wait for exit; investigate block reports.
• Kerberos ticket expired: kinit with keytab; re‑run plan/apply.
• Authorization denied: adjust policy or use admin principal; revert any partial changes using prior state.
• Quota already exceeded: either clean up data or temporarily increase quota under change control.
• Replication not converging: check DataNodes and capacity; revert to the previous factor if constrained.
• HA failover mid‑apply: retry after stabilization; keep batches short to minimize blast radius.

Tip: If snapshots are enabled on target directories, take a snapshot just before apply to simplify data rollback, for example: hdfs dfs -createSnapshot /data/projects/analytics pre_change_w32.

Conclusion

You do not need a heavyweight platform to gain reliability in HDFS changes. Start with two simple, valuable workflows: manage directories, quotas, and ACLs as code, and roll out replication changes with a clear plan and verification. Keep manifests in Git, run plan before apply, make changes idempotent, and bake in pre‑checks for HA, SafeMode, and authentication. Capture plan.out and apply.out on every run so rollback is fast and targeted. Once these patterns are routine, you can expand to more workflows (ingest landing zones, staging paths, or tiered retention) while keeping risk low and outcomes predictable.