Apache Hop Local Lab Setup with Practical Examples

Intro

Setting up a local lab for Apache Hop is the fastest way to learn the platform, prototype data pipelines, and debug transformations without touching shared environments. This guide walks through a minimal, reproducible setup on a single workstation, then shows practical examples of building and testing a pipeline. It is aimed at developers, DevOps engineers, and technical startup teams who want a safe sandbox for pipeline development.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. All examples use a local environment with no external dependencies, so you can follow along without access to a cluster or cloud account.

Version and Environment Inventory

Prerequisites

• Java 11 or 17: Apache Hop 2.x requires Java 11 or later. Use java -version to check. Example output: openjdk version "17.0.8" 2023-07-18.
• Enough memory: Hop runs comfortably with 2 GB allocated to the JVM. A workstation with 8 GB or more total RAM is recommended.
• Operating system: Windows, macOS, or Linux. Commands below are shown for Linux/macOS; adjust paths for Windows.
• Download Apache Hop: Get the latest stable binary distribution from the official website (e.g., apache-hop-client-2.9.0.zip).

Read-only Observation

Before changing anything, inspect the current state.

# Check Java version
java -version

# Check if Hop is already installed
which hop

# Check if a Hop home directory exists
ls -la ~/hop

Expected result: Java reports a supported version (11 or 17), which hop returns nothing if Hop is not installed, and the ~/hop directory does not exist. If Hop is already present, note its version and path before proceeding.

Smallest Justified Change

Extract the zip archive into a dedicated directory, e.g. ~/hop.

mkdir -p ~/hop
cd ~/hop
unzip ../apache-hop-client-2.9.0.zip
# Resulting layout: ~/hop/apache-hop-client-2.9.0/...

Verification

Run Hop's command-line tool to confirm it starts.

cd ~/hop/apache-hop-client-2.9.0
./hop-conf.sh --version

Expected output includes the Hop version, for example: Apache Hop 2.9.0.

Recovery

If hop-conf.sh fails, check Java installation and the JAVA_HOME environment variable.

# Set JAVA_HOME on Linux/macOS (adjust path to your JDK)
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
# Verify
$JAVA_HOME/bin/java -version

Safe Configuration Path

Configuration Files

Hop stores its main configuration in hop-config directory inside the Hop home. The key file is hop-config/metastore.xml, which contains connection definitions, cluster schemas, and other shared metadata. For a local lab, the default configuration from the distribution is sufficient. However, to avoid accidentally connecting to a production environment, isolate the lab by using a separate project directory.

Create a dedicated project folder for your work:

mkdir -p ~/hop-lab/projects/my-first-project

Environment Variables

Set HOP_HOME to point to your Hop installation. This variable is used by scripts and plugins.

export HOP_HOME=~/hop/apache-hop-client-2.9.0
# Verify
echo $HOP_HOME

Add this line to your shell profile (.bashrc, .zshrc, etc.) to make it permanent.

Database Connection (Optional)

For many examples, a local database is useful. Hop includes an embedded H2 database driver, so no external database is required. For a more realistic setup, you can use PostgreSQL or MySQL. If using PostgreSQL, install it locally and create a database for the lab:

# Example on Ubuntu
sudo apt install postgresql
sudo -u postgres createdb hoplab

Then in Hop, create a new database connection in the metadata with these settings:

• Connection name: LocalPG
• Connection type: PostgreSQL
• Hostname: localhost
• Database name: hoplab
• Port: 5432
• Username: hopuser (create this user)
• Password: use a placeholder, e.g. ChangeMe123!

Test the connection by clicking "Test" in the metadata editor. Expected result: "Connection to database [LocalPG] is correct."

Recovery: If the test fails, check that PostgreSQL is running (sudo systemctl status postgresql), the database exists, and the user has privileges. Correct the connection settings and retry.

Version-appropriate Commands

Hop provides several command-line tools:

• hop-run executes pipelines and workflows.
• hop-conf manages the configuration.
• hop-server starts the Hop server for remote execution.
• hop-encrypt encrypts passwords for use in configuration files.

For example, to encrypt a password for use in a connection:

$HOP_HOME/hop-encrypt.sh -k ~/.hop/security.key -p ChangeMe123!

Expected output: an encrypted string like Encrypted 2be98afc86aa7f2e4cb79ce309ed2ef9a. Use this encrypted string in the connection's password field instead of the plain text.

Building a First Pipeline: Practical Example

Let's create a simple pipeline that reads data from a CSV file, transforms it, and writes to another CSV file. This example demonstrates the core Hop concepts: transforms, hops, and fields.

Step 1: Create a Sample CSV File

Create a file input.csv in your project directory:

id,name,salary
1,Alice,70000
2,Bob,80000
3,Charlie,90000

Step 2: Create a New Pipeline

Launch the Hop GUI:

cd $HOP_HOME
./hop-gui.sh

In the GUI, click "New" -> "Pipeline". Save it as first-pipeline.hpl in your project directory.

Step 3: Add a CSV Input Transform

From the "Input" folder in the design palette, drag a "CSV file input" transform onto the canvas. Configure it:

• Filename: ~/hop-lab/projects/my-first-project/input.csv
• Delimiter: ,
• Header row present: check
• Click "Get fields" to populate the field names: id, name, salary.

Step 4: Add a Calculator Transform

Drag a "Calculator" transform from the "Transform" folder. Connect the CSV input to the calculator with a hop.

In the calculator, add a new field:

• New field: salary_with_bonus
• Calculation: salary + (salary * 0.1)

This adds a 10% bonus to the salary.

Step 5: Add a CSV Output Transform

Drag a "CSV file output" transform from the "Output" folder. Connect the calculator to it.

Configure the output:

• Filename: ~/hop-lab/projects/my-first-project/output.csv
• Delimiter: ,
• Include header row: check
• Click "Get fields" to map all fields.

Step 6: Run the Pipeline

Save the pipeline. Click the "Run" button (green play icon). Choose local execution. The pipeline will run and produce output.csv.

Expected Output

Open output.csv and verify:

id,name,salary,salary_with_bonus
1,Alice,70000,77000.0
2,Bob,80000,88000.0
3,Charlie,90000,99000.0

Verification

The pipeline run log should show no errors. In the GUI, the "Logging" tab displays each transform's status. The final status should be "Finished" with a green checkmark.

Verification and Diagnostics

Checking Logs

Hop writes logs to logs/ inside the Hop home directory. For the GUI, logs are shown in the execution results panel. For command-line execution, use:

$HOP_HOME/hop-run.sh -f ~/hop-lab/projects/my-first-project/first-pipeline.hpl -r local

The -r local option specifies the local run configuration. The output includes start and end timestamps, number of rows processed, and any errors.

Example successful output snippet:

2024/01/15 10:12:34 - first-pipeline - Start of run.
2024/01/15 10:12:35 - CSV file input.0 - Finished processing (I=3, O=0, R=0, W=3, U=0, E=0)
2024/01/15 10:12:35 - Calculator.0 - Finished processing (I=3, O=0, R=0, W=3, U=0, E=0)
2024/01/15 10:12:35 - CSV file output.0 - Finished processing (I=3, O=0, R=3, W=0, U=0, E=0)
2024/01/15 10:12:35 - first-pipeline - End of run.

If an error occurs, the log shows an ERROR line with a stack trace. For example, if the input file is missing:

2024/01/15 10:15:00 - CSV file input.0 - ERROR: Unable to open file '~/hop-lab/projects/my-first-project/missing.csv'

Using the Hop Server for Debugging

For more complex pipelines, you can run them on the Hop server and monitor execution remotely. Start the server:

$HOP_HOME/hop-server.sh -p 8080

Then in the GUI, create a new run configuration with "Remote Hop server" and specify localhost:8080. This allows you to watch the pipeline steps in real time.

Inspecting Metadata

Use the metadata explorer to verify connections and shared objects. In the GUI, click "Tools" -> "Metadata". Check that your database connection appears and that the encrypted password is used.

Failure Modes and Recovery

Common Failure: Java Version Mismatch

Symptom: hop-gui.sh fails to start with UnsupportedClassVersionError or Java version error.

Diagnosis: java -version shows Java 8 or lower.

Recovery: Install Java 11 or 17, and ensure JAVA_HOME points to it. Update your PATH if necessary.

Common Failure: File Permissions

Symptom: hop-conf.sh: Permission denied.

Diagnosis: The extracted files may not have executable permissions.

Recovery: Run chmod +x $HOP_HOME/*.sh to make all scripts executable.

Common Failure: Out of Memory

Symptom: Pipeline fails with java.lang.OutOfMemoryError: Java heap space.

Diagnosis: The default JVM heap size may be too small for large datasets.

Recovery: Increase the heap size by setting the HOP_OPTS environment variable before starting Hop:

export HOP_OPTS="-Xmx2048m"

Then restart Hop.

Common Failure: Incorrect Path in CSV Input

Symptom: Pipeline runs but produces zero rows, or errors with "File not found".

Diagnosis: Check the filename in the CSV input transform. Ensure absolute path or correct relative path is used.

Recovery: Edit the transform, correct the path, and rerun.

Operations Checklist

Use this checklist before and after making changes to the lab environment or pipelines.

Before Changes

• [ ] Record current Hop version: $HOP_HOME/hop-conf.sh --version.
• [ ] Record current Java version: java -version.
• [ ] Backup any configuration files you are about to modify (e.g., hop-config/metastore.xml).
• [ ] For database connections, record connection name, host, port, and database name. Do not record passwords.

After Changes

• [ ] Verify Hop starts: run $HOP_HOME/hop-gui.sh or $HOP_HOME/hop-conf.sh --version.
• [ ] For modified pipelines, run them in local mode and check logs for errors.
• [ ] Confirm expected output files are created and have correct content (e.g., compare row counts).
• [ ] Test recovery: if a failure occurred, document the exact error and the steps taken to fix it.

Ownership and Review Frequency

• Owner: The designated lab environment owner (e.g., Priya Shah, Engineering Lead) is accountable for maintaining the lab setup and ensuring the checklist is followed.
• Review frequency: The lab setup and backup processes should be reviewed monthly, or after any major Hop version upgrade.

Common Pitfalls and How to Avoid Them

Pitfall 1: Mixing Lab and Production Metadata

Why it happens: Developers may import production metadata into the lab to test with real connections.

How to avoid: Always use separate projects or separate Hop homes for lab and production. If you must copy, change all connection parameters to local placeholders and encrypt passwords with a lab-specific key.

Recovery: If you accidentally used a production connection, immediately revoke the credentials and change them in the production system. Remove the imported metadata from the lab.

Pitfall 2: Not Versioning Pipelines

Why it happens: Lab pipelines are often treated as throwaway, so they are not committed to version control.

How to avoid: Initialize a Git repository in your project directory and commit pipeline files (.hpl, .hwf) regularly.

cd ~/hop-lab/projects/my-first-project
git init
git add *.hpl
git commit -m "Initial pipeline"

Pitfall 3: Hardcoding Secrets in Transforms

Why it happens: For quick tests, developers may put plaintext passwords directly in transform configurations.

How to avoid: Use Hop's built-in variable system and encrypted values. Define variables in kettle.properties (located in $HOP_HOME/config) and reference them in transforms with ${VARIABLE_NAME}. Encrypt sensitive variables using hop-encrypt.

Recovery: Search for plaintext secrets in your project files using grep -r 'password' . and replace them with variables or encrypted strings.

Pitfall 4: Ignoring Hop Version Compatibility with Plugins

Why it happens: Users may install plugins from different Hop versions, causing class conflicts.

How to avoid: Only install plugins from the Hop Marketplace that match your Hop version. Always check the plugin's compatibility matrix.

Recovery: If a plugin causes startup failure, remove it from $HOP_HOME/plugins and restart Hop.

Conclusion

A local Apache Hop lab gives you a safe, controlled environment to build and test data pipelines. By following the version-scoped, observable practices in this guide, you can avoid common pitfalls and recover quickly from failures. Start with the simple CSV pipeline, then expand to database connections and more complex workflows. Remember to keep your lab isolated from production, version your work, and document every recovery step. With these habits, your Hop lab will become a reliable foundation for pipeline development.