Apache Hop CI/CD Automation: Practical Examples for Reliable Pipelines

Intro

Apache Hop is an open-source data integration platform that lets you design and run data pipelines visually. As teams scale their data operations, they need CI/CD (Continuous Integration and Continuous Delivery) practices to deploy pipelines reliably. This article provides practical examples for automating Apache Hop pipeline deployment, configuration, verification, and rollback.

We focus on developers, DevOps consultants, and technical startup teams who need operational safety. You will learn how to check your environment, make controlled changes, verify outcomes, and recover from failures. Every step includes concrete commands, expected output, failure signals, and recovery paths.

The goal is operational safety: observe before changing, limit the blast radius, avoid secrets in commands, verify results, and know how to roll back. We use placeholders like <HOP_HOME> and <ENVIRONMENT> instead of real credentials or production identifiers.

Version and Environment Inventory

Before automating anything, know what you have. This section covers the essential inventory commands and checks for Apache Hop CI/CD.

Installed Version and Deployment Topology

First, identify the Apache Hop version and how it is deployed (standalone, clustered, or containerized). Use the Hop CLI or check the installation directory.

Example command (read-only):

cd <HOP_HOME>
./hop-conf.sh --version

Expected output:

Apache Hop 2.1.0

If the version is older than 2.0, review upgrade notes before proceeding.

Deployment topology matters: if you run Hop Server (remote execution), check that the server is reachable.

Example command:

curl -s http://<HOP_SERVER_HOST>:8080/hop/status

Expected output:

{"status":"UP"}

Record the output and timestamp in your inventory document.

Prerequisites

Ensure you have:

• Java 11 or higher (Hop 2.x requires Java 11+).
• Environment variables set: HOP_HOME, HOP_CONFIG_FOLDER.
• Access to version control for pipelines and configurations.
• Git installed and configured.

Example check:

java -version

Expected output:

openjdk version "11.0.20" 2023-07-18

If not, install Java 11 before continuing.

Read-Only Observation

Always start with a read-only command to understand the current state. For example, list all projects in Hop:

./hop-run.sh -j list_projects -f <HOP_CONFIG_FOLDER>

Expected output:

Projects:
  - sales_etl
  - marketing_etl

This tells you which pipelines exist and their names.

Smallest Justified Change

When changing configuration, modify one item at a time. Example: update a database connection string in a shared project. First, view the current value:

./hop-conf.sh -p sales_etl -g <ENVIRONMENT> -c database_connection -a show

Expected output:

Current value: jdbc:postgresql://old-db-host:5432/sales

Then plan the change: update to jdbc:postgresql://new-db-host:5432/sales with a rollback plan.

Verification Signal

After any change, verify with a command that shows the new value:

./hop-conf.sh -p sales_etl -g <ENVIRONMENT> -c database_connection -a show

Expected output:

Current value: jdbc:postgresql://new-db-host:5432/sales

If the output does not match, revert to the previous value.

Safe Configuration Path

Configuration changes are common in CI/CD. This section shows how to make them safely using environment-specific configuration files and Hop's built-in tools.

Use Environment-Specific Configurations

Apache Hop supports configuration folders for different environments (dev, test, prod). Store connection details and variables in environment-specific files, not hardcoded in pipelines.

Example:

In <HOP_CONFIG_FOLDER>/environments/prod/config.xml, set a variable:

<hop-config>
  <variable>
    <name>DB_HOST</name>
    <value>prod-db.internal</value>
  </variable>
</hop-config>

In your pipeline, reference ${DB_HOST}. This keeps secrets out of your repository and allows per-environment overrides.

Change Management with Git

Version your pipelines and configurations in Git. For example, a repository structure:

hop-pipelines/
  projects/
    sales_etl/
      main.hpl
      transform.hpl
  config/
    dev/
      config.xml
    prod/
      config.xml

When you need to change a pipeline, create a branch:

git checkout -b feature/update-sales-connection

Make changes, test locally, then merge via pull request after review.

Applying Configuration Changes

After merging, apply the new configuration to the target environment.

Example command:

./hop-conf.sh -p sales_etl -g prod -c database_connection -a update -v "jdbc:postgresql://new-db-host:5432/sales"

Expected output:

Configuration updated successfully.

Then verify as shown earlier.

Blast Radius and Rollback

Before applying, determine the blast radius: which pipelines use this configuration? Use Hop's dependency checker:

./hop-run.sh -j list_dependencies -p sales_etl -f <HOP_CONFIG_FOLDER>

This lists all pipelines that depend on the configuration item.

To rollback, reverse the change:

./hop-conf.sh -p sales_etl -g prod -c database_connection -a update -v "jdbc:postgresql://old-db-host:5432/sales"

Test this recovery path in a staging environment before relying on it in production.

Verification and Diagnostics

After deploying a pipeline, verify it works as expected. This section covers functional testing and diagnostics.

Run a Pipeline in Test Mode

Apache Hop can run a pipeline and capture metrics without writing to the target (depending on pipeline design). For a simple test, run the pipeline with a limited number of rows.

Example command:

./hop-run.sh -j main -p sales_etl -f <HOP_CONFIG_FOLDER> -r "first=10"

Expected output:

Pipeline executed successfully. Rows processed: 10

If the pipeline fails, inspect the log file specified in the output.

Check Logs for Errors

Hop logs are typically in <HOP_HOME>/logs. Use grep to find errors:

grep -i "error" <HOP_HOME>/logs/hop.log | tail -20

Example output:

2024-01-15 10:30:45 ERROR Database connection failed: jdbc:postgresql://new-db-host:5432/sales

This indicates a connectivity issue. Fix the connection string or network access.

Validate Pipeline Metadata

Before running, validate the pipeline XML for syntax errors:

./hop-run.sh -j validate -p sales_etl -f <HOP_CONFIG_FOLDER>

Expected output:

Pipeline validation successful.

If not, the output will list the specific errors.

Monitor Hop Server

If using Hop Server, monitor its health and pipeline execution status:

curl -s http://<HOP_SERVER_HOST>:8080/hop/pipelines

Expected output (JSON):

[{"name":"sales_etl.main","status":"RUNNING","lastExecution":"2024-01-15T10:30:00Z"}]

This helps you confirm the pipeline is running as scheduled.

Failure Modes and Recovery

Failures happen. This section describes common failure modes and how to recover.

Configuration Error

Failure mode: Pipeline fails because a variable is undefined or wrong.

Diagnosis: Check logs for "Variable not found" or similar.

Example:

2024-01-15 11:00:00 ERROR Variable ${DB_HOST} not defined in environment prod

Recovery: Define the variable in the environment configuration or correct the pipeline parameter. Then rerun.

Connectivity Failure

Failure mode: Database connection fails due to network or credentials.

Diagnosis: Use ping, telnet, or database client to test connectivity. In Hop, the error message will show the connection URL.

Recovery: Fix network access or update credentials via hop-conf.sh. Verify with ./hop-conf.sh -a show.

Pipeline Logic Error

Failure mode: Pipeline runs but produces wrong results.

Diagnosis: Compare output counts and sample data with expected. Use Hop's built-in logging to capture intermediate rows.

Recovery: Fix the pipeline design, commit changes, and redeploy. Rollback to a previous version if needed.

Rollback Procedures

Always have a rollback plan. For pipeline changes, use Git to revert:

git revert <commit-hash>

For configuration, use the command to set the previous value (as shown earlier).

For a full environment rollback, redeploy the previous Hop configuration folder and restart Hop Server.

Example command to restart Hop Server (if using systemd):

sudo systemctl restart hop-server

Then verify status:

sudo systemctl status hop-server

Expected output:

● hop-server.service - Apache Hop Server
   Loaded: loaded (/etc/systemd/system/hop-server.service; enabled)
   Active: active (running) since Mon 2024-01-15 12:00:00 UTC; 5s ago

Operations Checklist

Use this checklist before and after any CI/CD operation. Fill in the concrete values for your environment.

Pre-Change Checklist

• [ ] Record current Apache Hop version: ./hop-conf.sh --version → output: Apache Hop 2.1.0
• [ ] Verify environment variables: echo $HOP_HOME → /opt/hop
• [ ] List current projects: ./hop-run.sh -j list_projects -f <HOP_CONFIG_FOLDER> → output: sales_etl, marketing_etl
• [ ] Identify target pipeline and its dependencies: ./hop-run.sh -j list_dependencies -p sales_etl -f <HOP_CONFIG_FOLDER> → output: main.hpl depends on transform.hpl
• [ ] Backup current configuration: cp -r <HOP_CONFIG_FOLDER> <BACKUP_LOCATION>
• [ ] Define blast radius: which pipelines/processes are affected?
• [ ] Prepare rollback commands (e.g., revert commit, restore config backup).

Change Execution

• [ ] Apply configuration change or deploy new pipeline.
• [ ] Run validation: ./hop-run.sh -j validate -p sales_etl -f <HOP_CONFIG_FOLDER> → expected: Pipeline validation successful.
• [ ] Run test execution: ./hop-run.sh -j main -p sales_etl -f <HOP_CONFIG_FOLDER> -r "first=10" → expected: Rows processed: 10
• [ ] Check logs for errors: grep -i "error" <HOP_HOME>/logs/hop.log | tail -20 → expected: no new errors.

Post-Change Verification

• [ ] Confirm new configuration value: ./hop-conf.sh -p sales_etl -g prod -c database_connection -a show → expected: jdbc:postgresql://new-db-host:5432/sales
• [ ] Monitor pipeline execution via Hop Server or logs: curl -s http://<HOP_SERVER_HOST>:8080/hop/pipelines → expected: status RUNNING
• [ ] Compare data output with expected (e.g., row counts, sample values).
• [ ] Update documentation and inventory records.

Rollback Execution (if needed)

• [ ] Revert Git commit: git revert <commit-hash>
• [ ] Restore configuration from backup: cp -r <BACKUP_LOCATION>/* <HOP_CONFIG_FOLDER>/
• [ ] Restart Hop Server: sudo systemctl restart hop-server
• [ ] Verify service status: sudo systemctl status hop-server → expected: active (running)
• [ ] Re-run verification checks to confirm rollback success.

Conclusion

Apache Hop CI/CD automation works only when each step is version-scoped, observable, and reversible. Copying commands without checking prerequisites and expected output leads to failures. Use the checklists and examples in this article to build a safe deployment pipeline for your data workflows.

Start with one low-risk verification: record the current state, run a documented check, compare results, and review dependencies. Then expand your automation incrementally.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.