Apache Hop pipelines and workflows represent critical data infrastructure for many organizations. When a metadata repository corrupts, a pipeline fails silently, or an environment migration goes sideways, the ability to restore to a known-good state determines recovery time. This guide covers practical backup and restore procedures for Apache Hop deployments, from standalone development instances to production Kubernetes clusters. You will learn what to back up, how to automate it, how to verify restore integrity, and how to recover when things go wrong. ## Understanding What Needs Protection Apache Hop stores its configuration, metadata, and runtime state across several distinct locations. Missing any of these creates restore gaps that surface only during an actual incident. Metadata Repository — The backbone of every Hop project. Whether you use a file-based repository (JSON files under a project directory) or a database repository (PostgreSQL, MySQL, MariaDB, or H2), this contains pipelines, workflows, transforms, variables, and connection definitions. File-based repos live in your project folder; database repos require schema-level dumps. Configuration Files — The hop-config.json file (typically in ~/.hop/ or $HOP_CONFIG_DIRECTORY ) holds GUI preferences, recent projects, and plugin configurations. Environment files ( hop-environments.json ) define development, test, and production contexts with their variable overrides. The projects/ directory contains project-specific project-config.json files that map logical names to physical paths. Runtime Artifacts — Log files, execution results, and temporary files in $HOP_LOG_DIRECTORY or the project's logs/ folder. While not strictly required for restore, these accelerate root-cause analysis after a failure. External Dependencies — JDBC drivers in plugins/databases/ , custom plugins in plugins/ , and any files referenced by relative paths in pipeline metadata (CSV inputs, Avro schemas, Python scripts). These live outside the core Hop installation but are essential for pipeline execution. Version Alignment — Record the exact Hop version ( hop-gui --version or hop-run --version ), Java version ( java -version ), and database repository schema version. A restore onto a mismatched version often fails silently or produces subtle data corruption. Run this inventory command on any Hop node to capture the baseline: #!/usr/bin/env bash # capture-hop-inventory.sh — run as the service account that owns the Hop installation set -euo pipefail HOP_HOME="${HOP_HOME:-/opt/hop}" HOP_CONFIG_DIR="${HOP_CONFIG_DIR:-$HOME/.hop}" PROJECT_DIR="${PROJECT_DIR:-/data/hop-projects}" TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ") OUT_DIR="/var/backups/hop/inventory/$TIMESTAMP" mkdir -p "$OUT_DIR" echo "=== Apache Hop Version ===" > "$OUT_DIR/versions.txt" "$HOP_HOME/hop-run" --version >> "$OUT_DIR/versions.txt" 2>&1 java -version >> "$OUT_DIR/versions.txt" 2>&1 echo "=== Config Directory ===" >> "$OUT_DIR/versions.txt" ls -la "$HOP_CONFIG_DIR" >> "$OUT_DIR/versions.txt" echo "=== Project Structure ===" >> "$OUT_DIR/versions.txt" find "$PROJECT_DIR" -maxdepth 3 -type f -name "*.json" | head -50 >> "$OUT_DIR/versions.txt" echo "=== Database Repository (if configured) ===" >> "$OUT_DIR/versions.txt" if -f "$HOP_CONFIG_DIR/hop-config.json" ; then grep -A5 -B5 "databaseRepository" "$HOP_CONFIG_DIR/hop-config.json" >> "$OUT_DIR/versions.txt" || true fi echo "Inventory saved to $OUT_DIR" ## Backup Strategies by Deployment Model ### Standalone or VM-Based Deployment For single-node or active-passive VM deployments, a filesystem snapshot combined with a database dump provides complete coverage. File-Based Repository Backup #!/usr/bin/env bash # backup-hop-file-repo.sh — daily cron at 02:00 UTC set -euo pipefail PROJECT_ROOT="/data/hop-projects" BACKUP_ROOT="/var/backups/hop/file-repo" RETENTION_DAYS=14 TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ") DEST="$BACKUP_ROOT/$TIMESTAMP" mkdir -p "$DEST" # Exclude logs and temp files; they bloat backups and restore stale state rsync -av --delete \ --exclude="*/logs/*" \ --exclude="*/tmp/*" \ --exclude="*.tmp" \ --exclude="*.bak" \ "$PROJECT_ROOT/" "$DEST/projects/" # Backup config directory separately (small, changes infrequently) rsync -av "$HOME/.hop/" "$DEST/config/" # Create manifest for verification find "$DEST" -type f -exec sha256sum {} + > "$DEST/manifest.sha256" # Prune old backups find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} + echo "Backup completed: $DEST" Database Repository Backup (PostgreSQL Example) #!/usr/bin/env bash # backup-hop-db-repo.sh — runs on the database host or via pg_dump remote set -euo pipefail DB_HOST="${DB_HOST:-localhost}" DB_PORT="${DB_PORT:-5432}" DB_NAME="${DB_NAME:-hop_repo}" DB_USER="${DB_USER:-hop_backup}" BACKUP_ROOT="/var/backups/hop/db-repo" RETENTION_DAYS=30 TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ") DEST="$BACKUP_ROOT/$TIMESTAMP" mkdir -p "$DEST" # Use --no-owner --no-privileges for portable restores; --format=custom enables parallel restore pg_dump -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" \ --no-owner --no-privileges --format=custom \ --file="$DEST/hop_repo.dump" "$DB_NAME" # Verify dump integrity pg_restore --list "$DEST/hop_repo.dump" > /dev/null # Complement with config backup rsync -av "$HOME/.hop/" "$DEST/config/" find "$DEST" -type f -exec sha256sum {} + > "$DEST/manifest.sha256" find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime +$RETENTION_DAYS -exec rm -rf {} + echo "Database backup completed: $DEST" Schedule both via systemd timers or cron. Test restores monthly — an untested backup is a liability. ### Kubernetes Deployment (Helm or Operator) In Kubernetes, Hop typically runs as a Deployment or StatefulSet with a PersistentVolumeClaim for the project directory and a separate database (CloudSQL, RDS, or in-cluster PostgreSQL) for the metadata repository. Velero-Based Backup (Recommended) # velero-hop-backup.yaml — apply weekly via CronJob apiVersion: velero.io/v1 kind: Schedule metadata: name: hop-weekly-backup namespace: velero spec: schedule: "0 2 * * 0" # Sundays 02:00 UTC template: includedNamespaces: - hop-prod labelSelector: matchLabels: app.kubernetes.io/name: hop snapshotVolumes: true ttl: 720h0m0s # 30 days storageLocation: default hooks: resources: - name: hop-db-dump includedNamespaces: - hop-prod labelSelector: matchLabels: app: hop-postgres pre: - exec: container: postgres command: - /bin/bash - -c - | pg_dump -U hop_backup -Fc hop_repo > /tmp/hop_repo.dump cat /tmp/hop_repo.dump onError: Fail timeout: 300s Manual PVC Backup (No Velero) #!/usr/bin/env bash # backup-hop-pvc.sh — run from a pod with the PVC mounted set -euo pipefail PVC_MOUNT="/data/hop-projects" BACKUP_BUCKET="s3://my-org-hop-backups/prod/" TIMESTAMP=$(date -u +"%Y%m%dT%H%M%SZ") # Sync project files to object storage aws s3 sync "$PVC_MOUNT" "$BACKUP_BUCKET$TIMESTAMP/projects/" \ --exclude "*/logs/*" --exclude "*/tmp/*" --delete # Dump database from sidecar or init container kubectl exec -n hop-prod deploy/hop-postgres -- \ pg_dump -U hop_backup -Fc hop_repo | \ aws s3 cp - "$BACKUP_BUCKET$TIMESTAMP/hop_repo.dump" echo "Kubernetes backup completed: $BACKUP_BUCKET$TIMESTAMP" Include the Hop version in backup metadata (Velero annotations or S3 tags) to prevent version-mismatch restores. ### Docker Compose Deployment # docker-compose.backup.yml — run via `docker compose -f docker-compose.yml -f docker-compose.backup.yml up hop-backup` services: hop-backup: image: alpine:3.20 volumes: - hop-projects:/data/hop-projects:ro - hop-config:/root/.hop:ro - ./backups:/backups environment: - DB_HOST=hop-postgres - DB_NAME=hop_repo - DB_USER=hop_backup - DB_PASSWORD_FILE=/run/secrets/db_password secrets: - db_password command: > /bin/sh -c " set -euo pipefail TIMESTAMP=$$(date -u +'%Y%m%dT%H%M%SZ') DEST=/backups/$$TIMESTAMP mkdir -p $$DEST rsync -av --exclude='*/logs/*' --exclude='*/tmp/*' /data/hop-projects/ $$DEST/projects/ rsync -av /root/.hop/ $$DEST/config/ pg_dump -h $$DB_HOST -U $$DB_USER -Fc $$DB_NAME > $$DEST/hop_repo.dump find $$DEST -type f -exec sha256sum {} + > $$DEST/manifest.sha256 echo 'Backup complete: '$$DEST " depends_on: - hop-postgres volumes: hop-projects: hop-config: secrets: db_password: external: true ## Restore Procedures with Verification A restore is not complete until you verify the restored artifacts execute correctly. ### File-Based Repository Restore #!/usr/bin/env bash # restore-hop-file-repo.sh — interactive, requires confirmation set -euo pipefail BACKUP_ROOT="/var/backups/hop/file-repo" PROJECT_ROOT="/data/hop-projects" CONFIG_DIR="$HOME/.hop" echo "Available backups:" ls -1dt "$BACKUP_ROOT"/*/ | head -10 | nl read -rp "Enter backup number to restore (1-10): " SELECTION SELECTED=$(ls -1dt "$BACKUP_ROOT"/*/ | sed -n "${SELECTION}p") if -z "$SELECTED" ; then echo "Invalid selection" exit 1 fi echo "Selected: $SELECTED" echo "Verifying manifest..." cd "$SELECTED" sha256sum -c manifest.sha256 --quiet || { echo "Manifest verification FAILED. Aborting." exit 1 } read -rp "Manifest OK. Stop Hop services and proceed? [y/N] " CONFIRM [[ "$CONFIRM" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 1; } # Stop services (adapt to your init system) systemctl stop hop-server || true # Backup current state as rollback point ROLLBACK="/var/backups/hop/pre-restore-$(date -u +%Y%m%dT%H%M%SZ)" mkdir -p "$ROLLBACK" rsync -av "$PROJECT_ROOT/" "$ROLLBACK/projects/" rsync -av "$CONFIG_DIR/" "$ROLLBACK/config/" # Restore rsync -av --delete "$SELECTED/projects/" "$PROJECT_ROOT/" rsync -av "$SELECTED/config/" "$CONFIG_DIR/" # Verify Hop can load the project "$HOP_HOME/hop-run" --version "$HOP_HOME/hop-gui" --help > /dev/null 2>&1 || true # Test a known pipeline TEST_PIPELINE="$PROJECT_ROOT/my-project/pipelines/daily-etl.hpl" if -f "$TEST_PIPELINE" ; then echo "Running smoke test on $TEST_PIPELINE..." "$HOP_HOME/hop-run" --file "$TEST_PIPELINE" --log-level Basic || { echo "SMOKE TEST FAILED. Rolling back..." rsync -av --delete "$ROLLBACK/projects/" "$PROJECT_ROOT/" rsync -av "$ROLLBACK/config/" "$CONFIG_DIR/" exit 1 } fi systemctl start hop-server echo "Restore completed and verified." ### Database Repository Restore #!/usr/bin/env bash # restore-hop-db-repo.sh — restores PostgreSQL repository set -euo pipefail DB_HOST="${DB_HOST:-localhost}" DB_PORT="${DB_PORT:-5432}" DB_NAME="${DB_NAME:-hop_repo}" DB_USER="${DB_USER:-hop_admin}" BACKUP_FILE="${1:-}" if -z "$BACKUP_FILE" || ! -f "$BACKUP_FILE" ; then echo "Usage: $0 /path/to/hop_repo.dump" exit 1 fi echo "Restoring $BACKUP_FILE to $DB_HOST:$DB_PORT/$DB_NAME" read -rp "This will DROP and RECREATE the database. Continue? [y/N] " CONFIRM [[ "$CONFIRM" =~ ^[Yy]$ ]] || exit 1 # Terminate active connections psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -c " SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = '$DB_NAME' AND pid <> pg_backend_pid(); " # Drop and recreate psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -c "DROP DATABASE IF EXISTS $DB_NAME;" psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d postgres -c "CREATE DATABASE $DB_NAME;" # Restore with parallel jobs for speed pg_restore -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -j 4 "$BACKUP_FILE" # Verify table count matches expected baseline TABLE_COUNT=$(psql -h "$DB_HOST" -p "$DB_PORT" -U "$DB_USER" -d "$DB_NAME" -tAc " SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public'; ") echo "Restored table count: $TABLE_COUNT" # Verify Hop connectivity "$HOP_HOME/hop-run" --help > /dev/null echo "Database restore completed." ### Kubernetes Restore (Velero) # Restore from Velero schedule velero restore create --from-schedule hop-weekly-backup --namespace hop-prod # Monitor velero restore get velero restore logs --follow # Verify pods are ready kubectl wait --for=condition=Ready pod -l app.kubernetes.io/name=hop -n hop-prod --timeout=300s # Run smoke test pipeline via hop-run in a temporary pod kubectl run hop-smoke-test --rm -i --restart=Never \ --image=my-org/hop:2.12.0 \ --env="HOP_PROJECT=my-project" \ --env="HOP_ENVIRONMENT=production" \ -- /opt/hop/hop-run --file /data/hop-projects/my-project/pipelines/daily-etl.hpl --log-level Basic ## Validation, Monitoring, and Failure Modes ### Automated Backup Verification Do not assume backups work. Run a weekly verification job that restores to a temporary location and executes a smoke test pipeline. #!/usr/bin/env bash # verify-hop-backup.sh — runs weekly via cron set -euo pipefail LATEST_BACKUP=$(ls -1dt /var/backups/hop/file-repo/*/ | head -1) VERIFY_DIR="/tmp/hop-verify-$(date -u +%Y%m%dT%H%M%SZ)" SMOKE_PIPELINE="pipelines/health-check.hpl" HOP_HOME="/opt/hop" mkdir -p "$VERIFY_DIR" rsync -av "$LATEST_BACKUP/projects/" "$VERIFY_DIR/projects/" rsync -av "$LATEST_BACKUP/config/" "$VERIFY_DIR/config/" # Verify manifest cd "$LATEST_BACKUP" sha256sum -c manifest.sha256 --quiet || { echo "VERIFICATION FAILED: manifest mismatch in $LATEST_BACKUP" exit 1 } # Run smoke test with isolated config export HOP_CONFIG_DIRECTORY="$VERIFY_DIR/config" export HOP_PROJECT_DIRECTORY="$VERIFY_DIR/projects/my-project" "$HOP_HOME/hop-run" --file "$VERIFY_DIR/projects/my-project/$SMOKE_PIPELINE" --log-level Basic # Cleanup rm -rf "$VERIFY_DIR" echo "Backup verification PASSED: $LATEST_BACKUP" Alert on failure via your monitoring stack (Prometheus Alertmanager, PagerDuty, Opsgenie). ### Common Failure Modes and Recovery
Failure ModeDetectionRecovery Action
Manifest checksum mismatchVerification job failsInvestigate backup corruption; restore from previous known-good backup; check disk health
Database restore version mismatchpg_restore warnings or Hop startup errorsAlign Hop version with repository schema version; run migration scripts if available
Missing JDBC drivers after restorePipeline fails with ClassNotFoundExceptionRestore plugins/databases/ from backup or re-download matching driver versions
Environment variable driftPipelines connect to wrong databaseCompare restored hop-environments.json with current; re-apply environment-specific variables
Partial PVC restore (K8s)Some project files missingVerify Velero volume snapshot completion; check PVC binding and storage class
Corrupted metadata repositoryHop GUI shows empty project or errors on openRestore from last verified backup; do not attempt manual JSON repair
### Rollback Procedure When a planned change (upgrade, migration, config update) causes regression: - Stop the affected Hop services. - Snapshot current state to a rollback directory (as shown in the restore script). - Restore from the last verified backup. - Verify with smoke test pipeline. - Document the failure cause in your incident tracker before re-attempting the change. Keep at least three verified backup generations. A single backup generation offers no protection if the failure occurred before the last backup. ## Operational Checklist Use this checklist during onboarding, quarterly reviews, and incident postmortems. Daily - [ ] Backup job completed successfully (check logs/monitoring) - [ ] Manifest verification passed - [ ] Backup age < 26 hours (for daily schedule) Weekly - [ ] Automated verification job passed (full restore + smoke test) - [ ] Backup storage capacity > 20% free - [ ] Review backup duration trend (alert if > 2x baseline) Monthly - [ ] Manual restore drill to staging environment - [ ] Verify database repository schema version matches Hop version - [ ] Confirm JDBC drivers and custom plugins included in backup - [ ] Test cross-region restore (if multi-region strategy exists) Quarterly - [ ] Full disaster recovery exercise: simulate primary region loss - [ ] Review and update retention policy against compliance requirements - [ ] Validate restore runbooks with new team members - [ ] Confirm encryption at rest and in transit for backup data On Every Hop Upgrade - [ ] Take pre-upgrade backup (tagged with version) - [ ] Verify backup includes all plugin directories - [ ] Test restore on same version in staging - [ ] After upgrade, run verification job against new version - [ ] Update runbooks if CLI flags or config paths changed ## Conclusion Reliable Apache Hop backup and restore requires protecting four distinct artifact classes: the metadata repository (file or database), configuration directories, runtime plugins and drivers, and version alignment metadata. Automate backups with manifest verification, schedule weekly restore drills that execute a smoke-test pipeline, and maintain at least three verified backup generations. Treat an untested backup as a gap, not a safety net. When an incident occurs — whether a corrupted repository, a failed upgrade, or a cluster loss — the difference between a 30-minute recovery and a multi-day outage is a verified, practiced restore procedure. Start this week by running the inventory script, configuring one automated backup with verification, and scheduling your first restore drill.