Apache Hop Basic Commands with Practical Examples

Intro

Apache Hop (Hop Orchestration Platform) is an open-source data integration and orchestration tool designed for building, running, and monitoring data pipelines and workflows. The Hop command-line interface (CLI) provides a set of utilities for managing projects, environments, pipelines, and workflows without the graphical interface. Operators and developers use these commands to automate tasks, diagnose issues, and maintain reliable data processing systems.

This article provides practical, copy-paste-ready examples of essential Apache Hop commands. It targets developers, DevOps engineers, and data engineers who need to operate Hop in production and test environments. Every command includes prerequisites, a concrete example, expected output, common failure signals, and recovery guidance.

The goal is operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery steps. We assume Hop is installed and configured according to the official documentation, and we emphasize version-appropriate commands using Hop 1.2.0 as the reference version.

Version and Environment Inventory

The first step in any operational task is to determine the exact version of Apache Hop and its runtime environment. This information is critical for troubleshooting, compatibility checking, and applying the correct syntax for commands.

Check Hop Version

Prerequisite: Hop binaries are available in your PATH or you are in the Hop installation directory.

Command:

hop -version

Expected output:

Apache Hop 1.2.0

What to look for: The version number should match the version you intend to use. If the command fails with command not found, your PATH is not set correctly. If an older version appears, you may have multiple installations.

Recovery: Verify the installation directory and ensure hop script is executable. On Linux/macOS, check with which hop. On Windows, check the hop.bat file location and PATH environment variable.

Check Java Version

Prerequisite: Java is installed and JAVA_HOME is set.

Command:

java -version

Expected output (example for Java 11):

openjdk version "11.0.12" 2021-07-20
OpenJDK Runtime Environment (build 11.0.12+7)
OpenJDK 64-Bit Server VM (build 11.0.12+7, mixed mode)

What to look for: Apache Hop 1.2.0 supports Java 11 and 17. If you see Java 8 or an unsupported version, Hop may fail to start.

Recovery: Install a supported JDK and set JAVA_HOME correctly. For example, on Linux:

export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export PATH=$JAVA_HOME/bin:$PATH

Check Environment Configuration

Hop uses environment configuration files to define connections and variables. To list available environments, run:

Command:

hop -list-environments

Expected output:

Available environments:
- local
- dev
- prod

What to look for: Ensure the environment you intend to use exists. If not, you may need to create it using hop -create-environment or the GUI.

Recovery: If an environment is missing, define it in the config directory or use the command to create it. For example:

hop -create-environment -name staging -variables staging-variables.json

Safe Configuration Path

Apache Hop allows you to manage configuration files for projects, environments, and metadata. Modifying configuration can affect all pipelines, so it should be done carefully.

Back Up Configuration Before Changes

Always back up the config directory before making changes.

Command (Linux/macOS):

tar -czf hop-config-backup-$(date +%Y%m%d).tar.gz config/

Command (Windows PowerShell):

Compress-Archive -Path config\* -DestinationPath hop-config-backup-$(Get-Date -Format yyyyMMdd).zip

What to look for: The archive file should be created without errors. Verify its size is reasonable.

Recovery: If a configuration change breaks Hop, restore the backup:

tar -xzf hop-config-backup-YYYYMMDD.tar.gz

Change a Configuration Property Safely

Suppose you need to change the log level from INFO to DEBUG for troubleshooting. The Hop configuration is stored in files like hop-config.json or log4j2.xml depending on version. For Hop 1.2.0, logging configuration is in config/log4j2.xml.

Step 1: Observe current setting

grep -n "Root level" config/log4j2.xml

Expected output:

<Root level="INFO">

Step 2: Make a copy of the file

cp config/log4j2.xml config/log4j2.xml.bak

Step 3: Edit the file Change level="INFO" to level="DEBUG".

Step 4: Verify change

grep -n "Root level" config/log4j2.xml

Expected output:

<Root level="DEBUG">

Step 5: Test Run a simple pipeline to see if logs are more detailed.

Recovery: If something goes wrong, restore from backup:

cp config/log4j2.xml.bak config/log4j2.xml

Avoid Hardcoding Secrets

Never put passwords or tokens directly in configuration files or pipeline parameters. Use environment variables or Hop's variable system.

Example: In config/environments/dev.json, define a variable for database password:

{
  "variables": {
    "DB_PASSWORD": "${env:DB_PASSWORD}"
  }
}

Then in your pipeline, use ${DB_PASSWORD}. When running, set the environment variable:

export DB_PASSWORD='your-secret-password'
hop -run -file my-pipeline.hpl -environment dev

This keeps secrets out of version control.

Verification and Diagnostics

After running a pipeline or workflow, you need to verify it executed correctly and diagnose any issues.

Run a Pipeline with Logging

The basic command to run a pipeline is:

Command:

hop -run -file /path/to/my-pipeline.hpl -environment dev -level Basic

Expected output: If successful, you will see log messages showing the progress and completion. Example snippet:

2023-03-15 14:30:22 - INFO - Pipeline started
2023-03-15 14:30:25 - INFO - Transformation completed
2023-03-15 14:30:25 - INFO - Pipeline finished successfully

If there is an error, you will see stack traces or error messages.

Check Pipeline Result Using Log

Often, you want to check the result of a pipeline programmatically. You can use the -capture option to write metrics to a file:

Command:

hop -run -file /path/to/my-pipeline.hpl -environment dev -capture metrics.csv

Expected files:

• metrics.csv containing performance metrics like rows processed, duration, etc.

Inspect metrics:

head metrics.csv

Example content:

StepName,Copynr,Read,Written,Input,Output,Updated,Rejected,Errors,Status
Table input,0,10000,0,10000,0,0,0,0,Finished
Table output,0,0,10000,0,10000,0,0,0,Finished

What to look for: The Errors column should be 0 for all steps. If any step has a non-zero value, investigate.

Recovery: Check the Hop logs in logs/ directory for detailed error messages.

Diagnose Connection Issues

If a pipeline fails due to a database connection error, you can test the connection using a simple pipeline or the hop -test-database command if available. In Hop 1.2.0, there is no built-in database test command, but you can run a pipeline that contains only a database connection and a dummy transformation.

Alternatively, use the Hop GUI to test connections.

Command to run a minimal pipeline with database:

hop -run -file test-db-connection.hpl -environment dev

Expected output: If connection succeeds, the pipeline logs will show no errors. If it fails, the error message will indicate the cause (e.g., wrong host, authentication failure).

Recovery: Check the connection details in the environment configuration, ensure network access, and verify credentials.

Failure Modes and Recovery

Understanding common failure modes helps you respond quickly and minimize downtime.

Pipeline Fails Immediately with ClassNotFoundError

Symptom: When running a pipeline, you see an error like:

java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

Cause: The JDBC driver for MySQL is not in the classpath.

Recovery:

1. Download the MySQL JDBC driver JAR from the official site.
2. Place it in the lib directory of your Hop installation.
3. Restart Hop and run the pipeline again.

Pipeline Hangs and Never Completes

Symptom: Pipeline starts but does not finish, no error messages.

Possible causes: A transformation is waiting for input that never comes, a network resource is slow, or there is a deadlock.

Diagnostic steps:

1. Check CPU and memory usage of the Hop process.
2. Look at the logs for the last activity.
3. Use jstack on the Java process to get a thread dump and identify stuck threads.

Command for thread dump:

jstack <pid> > threaddump.txt

Analyze threaddump.txt for blocked or waiting threads.

Recovery: Once the cause is identified, terminate the pipeline (Ctrl+C or kill) and fix the underlying issue (e.g., adjust timeouts, fix data flow).

Workflow Fails Due to Missing Variable

Symptom: Error like:

Variable ${my_var} is not defined

Cause: The variable was not provided via environment or parameter.

Recovery: Ensure the variable is defined in the environment configuration or passed with -param option.

Example passing parameter:

hop -run -file my-pipeline.hpl -param my_var=value

Recovery from a Failed Pipeline Run

If a pipeline fails partway through, you may need to restart from a checkpoint. Hop does not automatically checkpoint, but you can design pipelines with idempotent transformations (e.g., using upsert instead of insert).

Recovery procedure:

1. Identify the step where the failure occurred from logs.
2. Determine if any partial data was written.
3. Clean up partial data if necessary (using SQL or other means).
4. Re-run the pipeline after fixing the cause.

Operations Checklist

Use this checklist to ensure you follow best practices for operating Apache Hop.

• [ ] Verify version: Run hop -version and ensure it matches expectations.
• [ ] Check Java: Run java -version and confirm supported version (11 or 17 for Hop 1.2.0).
• [ ] Back up configuration: Before changes, archive the config directory.
• [ ] Test connection: If using external resources, test connectivity before running full pipelines.
• [ ] Run pipeline in a test environment first: Use a dedicated environment (e.g., dev) before production.
• [ ] Monitor logs: Tail logs during run: tail -f logs/hop.log
• [ ] Capture metrics: Use -capture to save performance data for post-run analysis.
• [ ] Secure secrets: Never hardcode passwords; use environment variables or Hop variables.
• [ ] Have a rollback plan: Know how to restore configuration or previous pipeline version.
• [ ] Document incidents: Record failure symptoms, cause, and recovery steps for future reference.

Conclusion

Apache Hop basic commands are essential for effective data pipeline management. By using the version and environment inventory commands, you establish a solid baseline. Safe configuration practices prevent accidental outages. Verification and diagnostics help you confirm successful runs and quickly identify issues. Understanding failure modes and having recovery procedures minimizes downtime. The operations checklist serves as a quick reference for daily tasks.

As a next step, choose one low-risk verification from this article, such as checking the Hop version or backing up your configuration. Then, run a simple pipeline with logging and capture metrics to become familiar with the command-line workflow.

Remember, a reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.