Apache Hop Backup and Restore with Practical Examples: Implementation Guide

Apache Hop pipelines and workflows represent critical data infrastructure for many organizations. When a metadata repository corrupts, a pipeline fails silently, or an environment migration goes sideways, the ability to restore to a known-good state determines recovery time. This guide covers practical backup and restore procedures for Apache Hop deployments, from standalone development instances to production Kubernetes clusters. You will learn what to back up, how to automate it, how to verify restore integrity, and how to recover when things go wrong.

Understanding What Needs Protection

Apache Hop stores its configuration, metadata, and runtime state across several distinct locations. Missing any of these creates restore gaps that surface only during an actual incident.

Metadata Repository — The backbone of every Hop project. Whether you use a file-based repository (JSON files under a project directory) or a database repository (PostgreSQL, MySQL, MariaDB, or H2), this contains pipelines, workflows, transforms, variables, and connection definitions. File-based repos live in your project folder; database repos require schema-level dumps.

Configuration Files — The hop-config.json file (typically in ~/.hop/ or $HOP_CONFIG_DIRECTORY) holds GUI preferences, recent projects, and plugin configurations. Environment files (hop-environments.json) define development, test, and production contexts with their variable overrides. The projects/ directory contains project-specific project-config.json files that map logical names to physical paths.

Runtime Artifacts — Log files, execution results, and temporary files in $HOP_LOG_DIRECTORY or the project's logs/ folder. While not strictly required for restore, these accelerate root-cause analysis after a failure.

External Dependencies — JDBC drivers in plugins/databases/, custom plugins in plugins/, and any files referenced by relative paths in pipeline metadata (CSV inputs, Avro schemas, Python scripts). These live outside the core Hop installation but are essential for pipeline execution.

Version Alignment — Record the exact Hop version (hop-gui --version or hop-run --version), Java version (java -version), and database repository schema version. A restore onto a mismatched version often fails silently or produces subtle data corruption.

Run this inventory command on any Hop node to capture the baseline:

#!/usr/bin/env bash
# capture-hop-inventory.sh — run as the service account that owns the Hop installation
set -euo pipefail

HOP_HOME="${HOP_HOME:-/opt/hop}"
HOP_CONFIG_DIR="${HOP_CONFIG_DIR:-$HOME/.hop}"
PROJECT_DIR="${PROJECT_DIR:-/data/hop-projects}"
TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ")
OUT_DIR="/var/backups/hop/inventory/$TIMESTAMP"

mkdir -p "$OUT_DIR"

echo "=== Apache Hop Version ===" > "$OUT_DIR/versions.txt"
"$HOP_HOME/hop-run" --version >> "$OUT_DIR/versions.txt" 2>&1
java -version >> "$OUT_DIR/versions.txt" 2>&1

echo "=== Config Directory ===" >> "$OUT_DIR/versions.txt"
ls -la "$HOP_CONFIG_DIR" >> "$OUT_DIR/versions.txt"

echo "=== Project Structure ===" >> "$OUT_DIR/versions.txt"
find "$PROJECT_DIR" -maxdepth 3 -type f -name "*.json" | head -50 >> "$OUT_DIR/versions.txt"

echo "=== Database Repository (if configured) ===" >> "$OUT_DIR/versions.txt"
if  -f "$HOP_CONFIG_DIR/hop-config.json" ; then
  grep -A5 -B5 "databaseRepository" "$HOP_CONFIG_DIR/hop-config.json" >> "$OUT_DIR/versions.txt" || true
fi

echo "Inventory saved to $OUT_DIR"

Backup Strategies by Deployment Model

Standalone or VM-Based Deployment

For single-node or active-passive VM deployments, a filesystem snapshot combined with a database dump provides complete coverage.

File-Based Repository Backup

#!/usr/bin/env bash
# backup-hop-file-repo.sh — daily cron at 02:00 UTC
set -euo pipefail

PROJECT_ROOT="/data/hop-projects"
BACKUP_ROOT="/var/backups/hop/file-repo"
RETENTION_DAYS=14
TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ")
DEST="$BACKUP_ROOT/$TIMESTAMP"

mkdir -p "$DEST"

# Exclude logs and temp files; they bloat backups and restore stale state
rsync -av --delete \
  --exclude="*/logs/*" \
  --exclude="*/tmp/*" \
  --exclude="*.tmp" \
  --exclude="*.bak" \
  "$PROJECT_ROOT/" "$DEST/projects/"

# Backup config directory separately (small, changes infrequently)
rsync -av "$HOME/.hop/" "$DEST/config/"

# Create manifest for verification
find "$DEST" -type f -exec sha256sum {} + > "$DEST/manifest.sha256"

# Prune old backups
find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} +

echo "Backup completed: $DEST"

Database Repository Backup (PostgreSQL Example)

#!/usr/bin/env bash
# backup-hop-db-repo.sh — runs on the database host or via pg_dump remote
set -euo pipefail

DB_HOST="${DB_HOST:-localhost}"
DB_PORT="${DB_PORT:-5432}"
DB_NAME="${DB_NAME:-hop_repo}"
DB_USER="${DB_USER:-hop_backup}"
BACKUP_ROOT="/var/backups/hop/db-repo"
RETENTION_DAYS=30
TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ")
DEST="$BACKUP_ROOT/$TIMESTAMP"

mkdir -p "$DEST"

# Use --no-owner --no-privileges for portable restores; --format=custom enables parallel restore
pg_dump -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" \
  --no-owner --no-privileges --format=custom \
  --file="$DEST/hop_repo.dump" "$DB_NAME"

# Verify dump integrity
pg_restore --list "$DEST/hop_repo.dump" > /dev/null

# Complement with config backup
rsync -av "$HOME/.hop/" "$DEST/config/"

find "$DEST" -type f -exec sha256sum {} + > "$DEST/manifest.sha256"
find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} +

echo "Database backup completed: $DEST"

Schedule both via systemd timers or cron. Test restores monthly — an untested backup is a liability.

Kubernetes Deployment (Helm or Operator)

In Kubernetes, Hop typically runs as a Deployment or StatefulSet with a PersistentVolumeClaim for the project directory and a separate database (CloudSQL, RDS, or in-cluster PostgreSQL) for the metadata repository.

Velero-Based Backup (Recommended)

# velero-hop-backup.yaml — apply weekly via CronJob
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: hop-weekly-backup
  namespace: velero
spec:
  schedule: "0 2 * * 0"  # Sundays 02:00 UTC
  template:
    includedNamespaces:
      - hop-prod
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: hop
    snapshotVolumes: true
    ttl: 720h0m0s  # 30 days
    storageLocation: default
    hooks:
      resources:
        - name: hop-db-dump
          includedNamespaces:
            - hop-prod
          labelSelector:
            matchLabels:
              app: hop-postgres
          pre:
            - exec:
                container: postgres
                command:
                  - /bin/bash
                  - -c
                  - |
                    pg_dump -U hop_backup -Fc hop_repo > /tmp/hop_repo.dump
                    cat /tmp/hop_repo.dump
                onError: Fail
                timeout: 300s

Manual PVC Backup (No Velero)

#!/usr/bin/env bash
# backup-hop-pvc.sh — run from a pod with the PVC mounted
set -euo pipefail

PVC_MOUNT="/data/hop-projects"
BACKUP_BUCKET="s3://my-org-hop-backups/prod/"
TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ")

# Sync project files to object storage
aws s3 sync "$PVC_MOUNT" "$BACKUP_BUCKET$TIMESTAMP/projects/" \
  --exclude "*/logs/*" --exclude "*/tmp/*" --delete

# Dump database from sidecar or init container
kubectl exec -n hop-prod deploy/hop-postgres -- \
  pg_dump -U hop_backup -Fc hop_repo | \
  aws s3 cp - "$BACKUP_BUCKET$TIMESTAMP/hop_repo.dump"

echo "Kubernetes backup completed: $BACKUP_BUCKET$TIMESTAMP"

Include the Hop version in backup metadata (Velero annotations or S3 tags) to prevent version-mismatch restores.

Docker Compose Deployment

# docker-compose.backup.yml — run via `docker compose -f docker-compose.yml -f docker-compose.backup.yml up hop-backup`
services:
  hop-backup:
    image: alpine:3.20
    volumes:
      - hop-projects:/data/hop-projects:ro
      - hop-config:/root/.hop:ro
      - ./backups:/backups
    environment:
      - DB_HOST=hop-postgres
      - DB_NAME=hop_repo
      - DB_USER=hop_backup
      - DB_PASSWORD_FILE=/run/secrets/db_password
    secrets:
      - db_password
    command: >
      /bin/sh -c "
        set -euo pipefail
        TIMESTAMP=$$(date -u +'%Y%m%dT%H%M%SZ')
        DEST=/backups/$$TIMESTAMP
        mkdir -p $$DEST
        rsync -av --exclude='*/logs/*' --exclude='*/tmp/*' /data/hop-projects/ $$DEST/projects/
        rsync -av /root/.hop/ $$DEST/config/
        pg_dump -h $$DB_HOST -U $$DB_USER -Fc $$DB_NAME > $$DEST/hop_repo.dump
        find $$DEST -type f -exec sha256sum {} + > $$DEST/manifest.sha256
        echo 'Backup complete: '$$DEST
      "
    depends_on:
      - hop-postgres
volumes:
  hop-projects:
  hop-config:
secrets:
  db_password:
    external: true

Restore Procedures with Verification

A restore is not complete until you verify the restored artifacts execute correctly.

File-Based Repository Restore

#!/usr/bin/env bash
# restore-hop-file-repo.sh — interactive, requires confirmation
set -euo pipefail

BACKUP_ROOT="/var/backups/hop/file-repo"
PROJECT_ROOT="/data/hop-projects"
CONFIG_DIR="$HOME/.hop"

echo "Available backups:"
ls -1dt "$BACKUP_ROOT"/*/ | head -10 | nl

read -rp "Enter backup number to restore (1-10): " SELECTION
SELECTED=$(ls -1dt "$BACKUP_ROOT"/*/ | sed -n "${SELECTION}p")

if  -z "$SELECTED" ; then
  echo "Invalid selection"
  exit 1
fi

echo "Selected: $SELECTED"
echo "Verifying manifest..."
cd "$SELECTED"
sha256sum -c manifest.sha256 --quiet || {
  echo "Manifest verification FAILED. Aborting."
  exit 1
}

read -rp "Manifest OK. Stop Hop services and proceed? [y/N] " CONFIRM
[[ "$CONFIRM" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 1; }

# Stop services (adapt to your init system)
systemctl stop hop-server || true

# Backup current state as rollback point
ROLLBACK="/var/backups/hop/pre-restore-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$ROLLBACK"
rsync -av "$PROJECT_ROOT/" "$ROLLBACK/projects/"
rsync -av "$CONFIG_DIR/" "$ROLLBACK/config/"

# Restore
rsync -av --delete "$SELECTED/projects/" "$PROJECT_ROOT/"
rsync -av "$SELECTED/config/" "$CONFIG_DIR/"

# Verify Hop can load the project
"$HOP_HOME/hop-run" --version
"$HOP_HOME/hop-gui" --help > /dev/null 2>&1 || true

# Test a known pipeline
TEST_PIPELINE="$PROJECT_ROOT/my-project/pipelines/daily-etl.hpl"
if  -f "$TEST_PIPELINE" ; then
  echo "Running smoke test on $TEST_PIPELINE..."
  "$HOP_HOME/hop-run" --file "$TEST_PIPELINE" --log-level Basic || {
    echo "SMOKE TEST FAILED. Rolling back..."
    rsync -av --delete "$ROLLBACK/projects/" "$PROJECT_ROOT/"
    rsync -av "$ROLLBACK/config/" "$CONFIG_DIR/"
    exit 1
  }
fi

systemctl start hop-server
echo "Restore completed and verified."

Database Repository Restore

#!/usr/bin/env bash
# restore-hop-db-repo.sh — restores PostgreSQL repository
set -euo pipefail

DB_HOST="${DB_HOST:-localhost}"
DB_PORT="${DB_PORT:-5432}"
DB_NAME="${DB_NAME:-hop_repo}"
DB_USER="${DB_USER:-hop_admin}"
BACKUP_FILE="${1:-}"

if  -z "$BACKUP_FILE" || ! -f "$BACKUP_FILE" ; then
  echo "Usage: $0 /path/to/hop_repo.dump"
  exit 1
fi

echo "Restoring $BACKUP_FILE to $DB_HOST:$DB_PORT/$DB_NAME"
read -rp "This will DROP and RECREATE the database. Continue? [y/N] " CONFIRM
[[ "$CONFIRM" =~ ^[Yy]$ ]] || exit 1

# Terminate active connections
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -c "
  SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = '$DB_NAME' AND pid <> pg_backend_pid();
"

# Drop and recreate
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -c "DROP DATABASE IF EXISTS $DB_NAME;"
psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -c "CREATE DATABASE $DB_NAME;"

# Restore with parallel jobs for speed
pg_restore -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -j 4 "$BACKUP_FILE"

# Verify table count matches expected baseline
TABLE_COUNT=$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -tAc "
  SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';
")
echo "Restored table count: $TABLE_COUNT"

# Verify Hop connectivity
"$HOP_HOME/hop-run" --help > /dev/null
echo "Database restore completed."

Kubernetes Restore (Velero)

# Restore from Velero schedule
velero restore create --from-schedule hop-weekly-backup --namespace hop-prod

# Monitor
velero restore get
velero restore logs <restore-name> --follow

# Verify pods are ready
kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=hop -n hop-prod --timeout=300s

# Run smoke test pipeline via hop-run in a temporary pod
kubectl run hop-smoke-test --rm -i --restart=Never \
  --image=my-org/hop:2.12.0 \
  --env="HOP_PROJECT=my-project" \
  --env="HOP_ENVIRONMENT=production" \
  -- /opt/hop/hop-run --file /data/hop-projects/my-project/pipelines/daily-etl.hpl --log-level Basic

Validation, Monitoring, and Failure Modes

Automated Backup Verification

Do not assume backups work. Run a weekly verification job that restores to a temporary location and executes a smoke test pipeline.

#!/usr/bin/env bash
# verify-hop-backup.sh — runs weekly via cron
set -euo pipefail

LATEST_BACKUP=$(ls -1dt /var/backups/hop/file-repo/*/ | head -1)
VERIFY_DIR="/tmp/hop-verify-$(date -u +%Y%m%dT%H%M%SZ)"
SMOKE_PIPELINE="pipelines/health-check.hpl"
HOP_HOME="/opt/hop"

mkdir -p "$VERIFY_DIR"
rsync -av "$LATEST_BACKUP/projects/" "$VERIFY_DIR/projects/"
rsync -av "$LATEST_BACKUP/config/" "$VERIFY_DIR/config/"

# Verify manifest
cd "$LATEST_BACKUP"
sha256sum -c manifest.sha256 --quiet || {
  echo "VERIFICATION FAILED: manifest mismatch in $LATEST_BACKUP"
  exit 1
}

# Run smoke test with isolated config
export HOP_CONFIG_DIRECTORY="$VERIFY_DIR/config"
export HOP_PROJECT_DIRECTORY="$VERIFY_DIR/projects/my-project"

"$HOP_HOME/hop-run" --file "$VERIFY_DIR/projects/my-project/$SMOKE_PIPELINE" --log-level Basic

# Cleanup
rm -rf "$VERIFY_DIR"
echo "Backup verification PASSED: $LATEST_BACKUP"

Alert on failure via your monitoring stack (Prometheus Alertmanager, PagerDuty, Opsgenie).

Common Failure Modes and Recovery

Rollback Procedure

When a planned change (upgrade, migration, config update) causes regression:

1. Stop the affected Hop services.
2. Snapshot current state to a rollback directory (as shown in the restore script).
3. Restore from the last verified backup.
4. Verify with smoke test pipeline.
5. Document the failure cause in your incident tracker before re-attempting the change.

Keep at least three verified backup generations. A single backup generation offers no protection if the failure occurred before the last backup.

Operational Checklist

Use this checklist during onboarding, quarterly reviews, and incident postmortems.

Daily

• [ ] Backup job completed successfully (check logs/monitoring)
• [ ] Manifest verification passed
• [ ] Backup age < 26 hours (for daily schedule)

Weekly

• [ ] Automated verification job passed (full restore + smoke test)
• [ ] Backup storage capacity > 20% free
• [ ] Review backup duration trend (alert if > 2x baseline)

Monthly

• [ ] Manual restore drill to staging environment
• [ ] Verify database repository schema version matches Hop version
• [ ] Confirm JDBC drivers and custom plugins included in backup
• [ ] Test cross-region restore (if multi-region strategy exists)

Quarterly

• [ ] Full disaster recovery exercise: simulate primary region loss
• [ ] Review and update retention policy against compliance requirements
• [ ] Validate restore runbooks with new team members
• [ ] Confirm encryption at rest and in transit for backup data

On Every Hop Upgrade

• [ ] Take pre-upgrade backup (tagged with version)
• [ ] Verify backup includes all plugin directories
• [ ] Test restore on same version in staging
• [ ] After upgrade, run verification job against new version
• [ ] Update runbooks if CLI flags or config paths changed

Conclusion

Reliable Apache Hop backup and restore requires protecting four distinct artifact classes: the metadata repository (file or database), configuration directories, runtime plugins and drivers, and version alignment metadata. Automate backups with manifest verification, schedule weekly restore drills that execute a smoke-test pipeline, and maintain at least three verified backup generations. Treat an untested backup as a gap, not a safety net. When an incident occurs — whether a corrupted repository, a failed upgrade, or a cluster loss — the difference between a 30-minute recovery and a multi-day outage is a verified, practiced restore procedure. Start this week by running the inventory script, configuring one automated backup with verification, and scheduling your first restore drill.