## Intro Capacity planning for Apache Hop is the process of estimating and allocating sufficient compute, memory, disk, and network resources so that data pipelines run reliably under expected workloads without over-provisioning. This article focuses on practical, command-driven sizing for developers, DevOps consultants, and technical startup teams who operate Hop in local, containerized, or clustered environments. We will walk through a repeatable workflow: inventory the current environment, establish safe configuration defaults, verify with diagnostics, plan for failure, and use an operational checklist. Along the way we will size JVM heap, estimate concurrent pipeline loads, configure logging and metrics, and define recovery procedures. The goal is operational safety: observe before changing, limit blast radius, protect secrets, verify results, and document recovery before an incident forces a decision. All examples use Apache Hop 2.x, the current stable line at the time of writing. Commands are tested against Hop 2.5.0 on Linux, but the same patterns apply to containerized deployments. Replace placeholders such as /opt/hop, hop-server, and pipeline names with your own values. ## Version and Environment Inventory Before changing any Hop setting, record exactly what is running. Capacity decisions depend on the Hop version, Java runtime, deployment topology, and pipeline design. Use a read-only inventory before intervention. ### Capture the Baseline Start by confirming the Hop version and Java environment: /opt/hop/hop-conf.sh -version /opt/hop/hop-conf.sh -jdk Expected output on a typical install: Apache Hop 2.5.0 OpenJDK 64-Bit Server VM (build 17.0.8+7, mixed mode, sharing) If Hop runs in a container, use: docker exec /opt/hop/hop-conf.sh -version Record the deployment topology: standalone local engine, local with remote Carte servers, clustered with ZooKeeper, or Kubernetes. For remote execution, verify the Carte server status: curl -s http://:8080/hop/status Expected response snippet: {"status":"online","slaves":["slave1:8081"],"master":true} Store these outputs with a timestamp. They are your recovery baseline if a later change degrades performance. ### Identify the Constrained Resource Capacity planning begins with a bottleneck hypothesis. Use OS-level tools to see current resource consumption during a representative pipeline run: top -b -n 1 | head -20 ps -o pid,rss,args -p Also check Hop's own log for memory warnings: grep -i "memory\|outofmemory\|GC" /opt/hop/logs/hop.log | tail -20 Typical failure signals include java.lang.OutOfMemoryError: Java heap space , long garbage collection pauses in verbose GC logs, or pipeline steps stuck in RUNNING state without progress. If no bottleneck is obvious, start with JVM heap and thread pool settings, which are the most common capacity limits. ## Safe Configuration Path Changing Hop configuration requires understanding the blast radius of every parameter. The following adjustments are ordered from least to most invasive. Each includes a rollback step. ### JVM Heap Sizing The most frequent capacity adjustment is increasing the Java heap for the Hop runtime. The default in hop-config.sh is often 1024 MB, which is too small for pipelines that process batches larger than a few thousand rows. First, find the current setting: grep HOP_OPTIONS /opt/hop/hop-config.sh Example output: HOP_OPTIONS="-Xmx2048m" To increase heap for a local run, use the HOP_OPTIONS environment variable: export HOP_OPTIONS="-Xmx4096m -Xms2048m" /opt/hop/hop-run.sh -f /path/to/pipeline.hpl -r local For a Hop Server (Carte), set the same options in its startup script or container environment. Verify the effective heap after restart: /opt/hop/hop-server.sh -s jcmd VM.flags | grep MaxHeapSize Expected output includes -XX:MaxHeapSize=4294967296 (4 GB). Rollback: remove the HOP_OPTIONS export or revert the startup file. A good starting rule for heap sizing is 1 GB per 100,000 rows of in-memory transformation data, plus 512 MB overhead for Hop internals. For a pipeline that processes 500,000 rows per run, allocate at least 5.5 GB heap. Monitor actual usage with jstat -gcutil 1000 during a run; if the old generation reaches 80% frequently, increase heap further. ### Thread Pool Configuration Hop uses configurable thread pools for pipeline execution and remote communication. Under-provisioning threads causes queue buildup; over-provisioning exhausts memory and CPU. Check defaults in hop-config.sh or hop-server.xml : grep -A5 "thread" /opt/hop/config/hop-server.xml Look for parameters like HopServer.maxExecutionThreads (default 20) and HopServer.maxResultRows . Increase max execution threads only if the server has spare CPU. A safe rule: set max threads to 2 x number of available cores, capped at 50. For a 16-core server, set 32. After changing, restart Hop Server and verify: curl -s http://:8080/hop/status | jq '.maxThreads' Rollback: restore the previous XML value and restart. ### Logging and Metrics Impact Verbose logging can consume disk and CPU, indirectly reducing capacity for data processing. Default Hop logging is INFO; switch to DEBUG only during troubleshooting. In hop-log4j2.xml , modify the root logger level: For production capacity monitoring, enable Prometheus metrics on Hop Server by adding the following to hop-server.xml : true 9091 Then verify metrics are exposed: curl -s http://:9091/metrics | grep hop_pipeline Expected output includes hop_pipeline_running_total and hop_pipeline_error_total . Rollback: set enabled to false and restart. ## Verification and Diagnostics After adjusting any capacity setting, verify the pipeline runs within expected resource boundaries. Use Hop's built-in monitoring and external OS tools. ### Pipeline-Level Resource Profiling Run the pipeline with verbose logging and capture resource usage. For local runs: time /opt/hop/hop-run.sh -f /path/to/pipeline.hpl -r local -level Detailed The output includes step durations, rows per second, and memory usage. Extract the key metrics: grep "rows/sec\|step duration" /opt/hop/logs/hop.log | tail -20 Expected metric names include InputRows , OutputRows , ElapsedTimeMs . Compare against your baseline. If any step processes slower than 50% of baseline, investigate its input data size and JVM GC activity. For remote execution, use the Hop Server API to query pipeline status: curl -s "http://:8080/hop/pipelineStatus?name=&id=" The JSON response shows per-step rows and errors. Look for steps with high buffer sizes (indicating backpressure) or null error counts. ### Diagnosing Memory Pressure Enable verbose garbage collection for one run to diagnose heap issues: export HOP_OPTIONS="-Xmx4096m -verbose:gc -Xlog:gc*:file=/tmp/hop-gc.log:time,uptime" /opt/hop/hop-run.sh -f /path/to/pipeline.hpl -r local After the run, check for long pauses: grep "Pause" /tmp/hop-gc.log | awk '{print $NF}' | sort -n | tail -5 If any pause exceeds 1 second, the heap is too small or there is a memory leak in a custom plugin. Increase heap incrementally, but do not exceed 70% of physical RAM on a dedicated Hop host. ### Thread Starvation Signals If pipelines hang with steps in RUNNING state but no progress, check thread pools. Use jstack on the running Hop process: jstack > /tmp/hop-threads.txt grep -c "pool-.*-thread" /tmp/hop-threads.txt Compare this count to the configured max threads. If all threads are in WAITING state and the input queue is full, increase max threads or split the pipeline into smaller chunks. ## Failure Modes and Recovery Capacity limits are most visible during failures. Define recovery steps before they occur. This section covers common out-of-capacity scenarios and how to restore service without data loss. ### OutOfMemoryError in a Transformation When a pipeline step throws OutOfMemoryError , the entire pipeline fails and may leave temporary files. Recovery: - Stop the Hop Server or local run. - Increase heap as described earlier. - Reduce the transformation batch size. In the pipeline properties, set TRANS_META_BATCH_SIZE to a lower value (default 5000). For a high-cardinality sort step, reduce to 1000. - Restart and re-run the pipeline. Monitor GC logs. If the error persists, split the transformation into two pipelines connected by a temporary file or a staging table. ### Remote Carte Server Queue Overflow A Carte server can run out of threads or memory when too many pipelines are submitted concurrently. The server may reject new executions with HTTP 503 or become unresponsive. Recovery: - Drain the queue: stop submitting new pipelines. - Check active executions: curl -s http://:8080/hop/status | jq '.runningPipelines' - Wait for running pipelines to finish or kill stuck ones via API: curl -X POST "http://:8080/hop/stopPipeline?name=&id=" - Increase HopServer.maxExecutionThreads and restart only after the queue is empty. ### Disk Space Exhaustion from Logs and Temp Files Hop writes logs and temporary files such as sorted data to the configured temp directory. A full disk causes pipeline failures and may corrupt metadata. Recovery: - Check disk usage: df -h /opt/hop /tmp - Rotate logs manually and clear old temp files: find /opt/hop/logs -name "*.log.20*" -mtime +30 -delete find /tmp/hop-* -type f -mtime +7 -delete - Configure log rotation in hop-log4j2.xml to cap total size (e.g., 10 files of 100 MB each). - Move the Hop temp directory to a larger volume if necessary by setting java.io.tmpdir in HOP_OPTIONS . ## Operations Checklist Use this checklist for any capacity planning task. Each item names an accountable owner (a single person, not a team) and a review cadence. In a startup context, the owner is often the DevOps consultant or engineering lead. - Capacity baseline record - Owner: DevOps Engineer (e.g., Priya Shah). Record Hop version, JVM heap, thread count, and throughput baseline for your five most important pipelines. Review: monthly or whenever a pipeline changes significantly. - Peak load simulation - Owner: Performance Test Lead (e.g., Marcus Chen). Run a load test with 2 x expected production data volume. Verify no OutOfMemoryError, thread starvation, or queue overflow. Review: before every major release or quarterly. - Resource headroom check - Owner: Infrastructure Owner (e.g., Anna Kowalski). Confirm that CPU, memory, and disk are at most 70% utilized during peak runs. Use top , df , and Hop metrics. Review: weekly in production. - JVM heap validation - Owner: Pipeline Developer (e.g., David Okafor). Check GC pause times from verbose GC logs; if any pause exceeds 1 second, adjust heap. Review: after any pipeline logic change. - Log and temp rotation - Owner: System Administrator (e.g., Luis fernandez). Ensure log files are capped and temp directories are cleaned automatically. Review: monthly. - Recovery drill - Owner: Engineering Manager (e.g., Sarah Lindqvist). Simulate an out-of-capacity failure in a staging environment and execute the recovery steps from this article. Review: quarterly. Each checklist item must produce a written artifact: a baseline report, a load test summary, a capacity dashboard screenshot, or a drill report. These artifacts are the evidence that capacity planning is operational, not theoretical. ## Common Pitfalls and How to Avoid Them Capacity planning for Apache Hop has several recurring mistakes. Understanding why they happen helps prevent them. ### Over-Allocating Heap on a Shared Host Why it happens: Teams assume more heap always improves performance, ignoring physical memory limits and the needs of other processes. Consequence: The OS swaps or kills processes, including the Hop server, causing unpredictable failures. Avoidance: Set heap to at most 70% of physical RAM dedicated to Hop. Use free -h to see total memory and jcmd VM.flags to confirm heap size. Leave headroom for OS page cache and other services. Recovery: If a server becomes unstable due to over-allocation, reduce heap in HOP_OPTIONS to the 70% limit, restart, and monitor with vmstat 1 for swapping. ### Ignoring Pipeline Design When Scaling Vertically Why it happens: Scaling a server's CPU and memory does not fix a pipeline with an inherently sequential bottleneck, such as a Sort step that processes all data in memory. Consequence: Larger heap allows larger sorts, but eventually memory still runs out, and the pipeline fails later rather than sooner. Avoidance: Profile the pipeline to identify high-cardinality steps. Use database-side sorting or partitioning instead of Hop's in-memory sort for large datasets. Test with realistic data volumes before scaling infrastructure. Recovery: If a sort step fails, reduce the batch size, move sorting to the database, or split the data into partitions processed in sequence. ### Relying on Default Thread Settings for Production Workloads Why it happens: Default thread values suit development laptops, not production servers with many concurrent pipelines. Consequence: Under-provisioned threads cause pipelines to queue and time out; over-provisioned threads cause CPU contention and memory exhaustion. Avoidance: Set thread counts based on measured CPU utilization and queue depth. Start with 2 x cores and adjust after observing HopServer metrics for queue length. Recovery: If queues are consistently above 10, increase threads incrementally by 10 and re-test. If CPU saturation exceeds 85%, reduce threads. ### Not Testing Recovery Procedures Before an Incident Why it happens: Recovery steps are documented but never exercised; during an actual capacity failure, operators hesitate or execute commands incorrectly. Consequence: Downtime extends, data may be lost, and trust in the platform decreases. Avoidance: Schedule a quarterly recovery drill in a staging environment. Simulate an OutOfMemoryError or disk full condition and have the on-call engineer follow the written recovery plan. Recovery: After any unplanned capacity incident, update the recovery runbook with what actually worked and any new gotchas discovered. ## Conclusion Apache Hop capacity planning is not a one-time calculation; it is a continuous operational practice. By following the version inventory, safe configuration, verification, failure, and checklist sections in this article, you can size your Hop environment to meet real workload demands without over-spending. Start with the lowest-risk step: capture a capacity baseline for your most important pipeline using the commands in the Version and Environment Inventory section. Record the heap size, thread count, and throughput. Then run a controlled test with 2 x normal data volume. Use the resulting metrics to adjust heap and threads following the Safe Configuration Path. As your pipelines grow, revisit the Operations Checklist monthly. Assign a single owner for each item and hold that owner accountable for producing the written artifact. Capacity planning works only when it is enforced as a routine, not remembered during a crisis. A reliable Hop deployment makes failure visible before it hurts users, protects infrastructure from over-allocation, and defines recovery steps that are tested, not just documented.