## Intro Bash is still the glue of many CI/CD pipelines. It is available everywhere, fast to iterate, and ideal for orchestrating tools like Git, Docker, and Kubernetes. This guide shows a production-ready approach to Bash in CI/CD: strict shell safety, clean logging, modular functions, retries and timeouts, validation before deploy, health verification, and fast rollbacks. You will also see GitHub Actions and GitLab CI examples you can copy into your repos today. What you will build: - A reusable Bash library (strict modes, logging, traps, retries, timeouts, locks) - Scripts for lint, test, build, validate, deploy, health-check, and rollback - CI pipelines for GitHub Actions and GitLab CI - Practical rollback patterns for Kubernetes and Docker Compose Recommended repo layout: . ├─ scripts/ │ ├─ lib.sh │ ├─ lint.sh │ ├─ test.sh │ ├─ build.sh │ ├─ validate.sh │ ├─ deploy.sh │ ├─ healthcheck.sh │ └─ rollback.sh ├─ docker/ │ └─ compose.yaml ├─ k8s/ │ ├─ deployment.yaml │ └─ service.yaml └─ app/ └─ ... ## 1) Safe Bash foundation (scripts/lib.sh) Use strict modes, predictable word splitting, defensive error handling, and structured logging. #!/usr/bin/env bash set -euo pipefail IFS=$'\n\t' # Colors only if stdout is a TTY if [ -t 1 ]; then GREEN='\033[0;32m'; YELLOW='\033[0;33m'; RED='\033[0;31m'; NC='\033[0m' else GREEN=''; YELLOW=''; RED=''; NC='' fi log() { printf "%sINFO%s %s\n" "$GREEN" "$NC" "$*"; } warn() { printf "%sWARN%s %s\n" "$YELLOW" "$NC" "$*"; } err() { printf "%sERROR%s %s\n" "$RED" "$NC" "$*" 1>&2; } fail(){ err "$*"; exit 1; } require() { command -v "$1" >/dev/null 2>&1 || fail "Missing command: $1"; } assert_env(){ for v in "$@"; do [ -n "${!v-}" ] || fail "Missing env var: $v"; done; } # Basic error context _on_error() { local ec=$? line=${1:-} err "Command failed at line $line (exit=$ec). Enable trace with: export TRACE=1" } # Optional tracing for debugging [ "${TRACE-}" = "1" ] && set -x trap '_on_error $LINENO' ERR # Run CMD with retries with_retry() { local attempts=${1:-5} sleep_s=${2:-2}; shift 2 local n=1 until "$@"; do if [ $n -ge "$attempts" ]; then err "Retry failed after $attempts attempts: $*"; return 1 fi warn "Attempt $n failed. Retrying in ${sleep_s}s..."; sleep "$sleep_s"; n=$((n+1)) done } # Timebox a command if 'timeout' exists timebox() { local seconds=$1; shift if command -v timeout >/dev/null 2>&1; then timeout "$seconds" "$@" else warn "timeout not found; running without time limit" "$@" fi } # Serialize critical sections lock_run() { local lockfile=$1; shift require flock exec 9>"$lockfile" flock -n 9 || fail "Another process holds $lockfile" "$@" } # Simple on-exit cleanup hook _cleanup_funcs=() on_exit() { _cleanup_funcs+=("$*"); } trap 'for f in "${_cleanup_funcs[@]}"; do eval "$f" || true; done' EXIT Key points: - set -euo pipefail and IFS reduce footguns and hidden failures. - trap with a custom handler gives actionable error context. - with_retry and timebox protect flaky or long-running operations. - lock_run serializes deploys to avoid race conditions. ## 2) Lint and test scripts Lint your Bash and run fast unit checks. Replace the test snippet with your language’s test runner if needed. scripts/lint.sh #!/usr/bin/env bash set -euo pipefail . "$(dirname "$0")/lib.sh" log "Linting bash scripts" find . -type f -name "*.sh" -print0 | xargs -0 -I{} bash -n {} if command -v shellcheck >/dev/null 2>&1; then log "Running shellcheck" find . -type f -name "*.sh" -print0 | xargs -0 shellcheck -x else warn "shellcheck not found; skipping static analysis" fi scripts/test.sh #!/usr/bin/env bash set -euo pipefail . "$(dirname "$0")/lib.sh" log "Running unit tests" add() { echo $(( $1 + $2 )); } assert_eq(){ [ "$1" = "$2" ] || fail "assert_eq failed: got=$1 want=$2"; } assert_eq "$(add 2 3)" 5 log "Tests passed" ## 3) Build an artifact Produce a Docker image when Docker is available; otherwise create a tarball so downstream steps still have a stable artifact. scripts/build.sh #!/usr/bin/env bash set -euo pipefail . "$(dirname "$0")/lib.sh" APP_NAME=${APP_NAME:-demoapp} IMAGE_REPO=${IMAGE_REPO:-example/demoapp} GIT_SHA=$(git rev-parse --short HEAD 2>/dev/null || echo local) TAG=${TAG:-$GIT_SHA} ART_DIR=${ART_DIR:-dist} mkdir -p "$ART_DIR" if command -v docker >/dev/null 2>&1; then log "Building Docker image $IMAGE_REPO:$TAG" docker build -t "$IMAGE_REPO:$TAG" -f docker/Dockerfile . echo "$IMAGE_REPO:$TAG" > "$ART_DIR/image.tag" else warn "docker not found; creating tarball artifact instead" tar -czf "$ART_DIR/$APP_NAME-$TAG.tgz" app/ echo "$ART_DIR/$APP_NAME-$TAG.tgz" > "$ART_DIR/artifact.path" fi ## 4) Validate before you deploy Validate manifests and compose configs in CI to fail early. scripts/validate.sh #!/usr/bin/env bash set -euo pipefail . "$(dirname "$0")/lib.sh" STAGE=${STAGE:-staging} COMPOSE_FILE=${COMPOSE_FILE:-docker/compose.yaml} log "Validating deployment manifests for $STAGE" if command -v kubectl >/dev/null 2>&1 && [ -d k8s ]; then require kubectl kubectl apply --dry-run=server -f k8s/ kubectl diff -f k8s/ || true elif command -v docker >/dev/null 2>&1 && [ -f "$COMPOSE_FILE" ]; then require docker docker compose -f "$COMPOSE_FILE" config >/dev/null else warn "No k8s or compose config found to validate" fi ## 5) Deploy safely and verify health Deploy with a lock to prevent overlapping runs, then verify a health endpoint before marking success. scripts/deploy.sh #!/usr/bin/env bash set -euo pipefail . "$(dirname "$0")/lib.sh" STAGE=${STAGE:-staging} HEALTH_URL=${HEALTH_URL:-http://localhost:8080/healthz} LOCKFILE=${LOCKFILE:-/tmp/deploy.lock} _do_deploy() { log "Starting deploy to $STAGE" if command -v kubectl >/dev/null 2>&1 && [ -d k8s ]; then kubectl apply -f k8s/ if kubectl get deploy >/dev/null 2>&1; then for d in $(kubectl get deploy -o name); do log "Waiting for $d to roll out" with_retry 30 2 kubectl rollout status "$d" done fi elif command -v docker >/dev/null 2>&1 && [ -f docker/compose.yaml ]; then docker compose -f docker/compose.yaml up -d --build else warn "No deploy target detected; skipping apply" fi "$(dirname "$0")/healthcheck.sh" "$HEALTH_URL" 60 2 log "Deployment verified" } lock_run "$LOCKFILE" _do_deploy scripts/healthcheck.sh #!/usr/bin/env bash set -euo pipefail . "$(dirname "$0")/lib.sh" URL=${1:-http://localhost:8080/healthz} ATTEMPTS=${2:-30} SLEEP=${3:-2} require curl with_retry "$ATTEMPTS" "$SLEEP" bash -c "curl -fsS --max-time 2 '$URL' >/dev/null" ## 6) Promote and rollback Record the deployed version so rollbacks are predictable. scripts/rollback.sh #!/usr/bin/env bash set -euo pipefail . "$(dirname "$0")/lib.sh" STAGE=${STAGE:-staging} PREV_FILE=${PREV_FILE:-.previous_version} CURR_FILE=${CURR_FILE:-dist/image.tag} COMPOSE_FILE=${COMPOSE_FILE:-docker/compose.yaml} promote() { [ -f "$CURR_FILE" ] || fail "Missing current version file: $CURR_FILE" cp "$CURR_FILE" "$PREV_FILE" log "Promoted version $(cat "$CURR_FILE") for $STAGE" } rollback() { [ -f "$PREV_FILE" ] || fail "No previous version recorded" local prev repo tag prev=$(cat "$PREV_FILE") repo=${prev%:*} tag=${prev#*:} if command -v kubectl >/dev/null 2>&1 && kubectl get deploy >/dev/null 2>&1; then for d in $(kubectl get deploy -o name); do log "Rolling back $d to image $prev" kubectl set image "$d" "*=$prev" --record with_retry 30 2 kubectl rollout status "$d" done elif command -v docker >/dev/null 2>&1 && [ -f "$COMPOSE_FILE" ]; then # Requires compose.yaml to use image: "${IMAGE_REPO}:${TAG}" or "${IMAGE}:${TAG}" IMAGE_REPO="$repo" TAG="$tag" docker compose -f "$COMPOSE_FILE" up -d log "Compose services updated to $repo:$tag" else fail "No supported platform for rollback" fi } case "${1:-}" in promote) promote ;; rollback) rollback ;; *) echo "Usage: $0 {promote|rollback}"; exit 2 ;; esac Tip: Kubernetes also supports kubectl rollout undo deploy/ to revert to the previous ReplicaSet without managing image tags. Using explicit tags plus a recorded file keeps history portable across clusters and runners. ## 7) CI integration GitHub Actions (/.github/workflows/ci.yaml): name: ci on: push: branches: [ main ] jobs: build-test-deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Lint run: scripts/lint.sh - name: Test run: scripts/test.sh - name: Build run: scripts/build.sh - name: Validate manifests env: STAGE: staging run: scripts/validate.sh - name: Deploy to staging and verify env: STAGE: staging HEALTH_URL: https://example-staging.local/healthz run: scripts/deploy.sh - name: Record version for rollback run: scripts/rollback.sh promote GitLab CI (/.gitlab-ci.yml): stages: [lint, test, build, validate, deploy] variables: GIT_STRATEGY: fetch lint: stage: lint image: alpine:3.20 before_script: ["apk add --no-cache bash findutils shellcheck"] script: - chmod +x scripts/*.sh - scripts/lint.sh unit_test: stage: test image: alpine:3.20 before_script: ["apk add --no-cache bash"] script: - chmod +x scripts/*.sh - scripts/test.sh build: stage: build image: docker:26 services: - docker:26-dind variables: DOCKER_HOST: tcp://docker:2375 DOCKER_TLS_CERTDIR: "" script: - chmod +x scripts/*.sh - scripts/build.sh artifacts: paths: - dist/ validate: stage: validate image: bitnami/kubectl:1.29 script: - chmod +x scripts/*.sh - STAGE=staging scripts/validate.sh deploy_staging: stage: deploy image: bitnami/kubectl:1.29 when: manual script: - chmod +x scripts/*.sh - STAGE=staging HEALTH_URL=https://example-staging.local/healthz scripts/deploy.sh - scripts/rollback.sh promote ## 8) Comparison tables Pipeline stages and goals:
StageScriptGoalFails if…
Lintscripts/lint.shCatch syntax/style issuesAny bash -n or shellcheck error
Testscripts/test.shVerify core behaviorAssertions fail
Buildscripts/build.shProduce image/tarballDocker build or tar creation fails
Validatescripts/validate.shDry-run manifests/configInvalid schema or bad references
Deployscripts/deploy.shApply + wait + health checkRollout or health endpoint fails
Promoterollback.sh promoteRecord current versiondist/image.tag missing
Rollbackrollback.sh rollbackRevert quicklyNo previous version or platform error
Rollback approaches:
PlatformMethodProsCons
Kuberneteskubectl rollout undoFast, uses previous ReplicaSetLimited to last ReplicaSet
KubernetesSet image to recorded tagExplicit, auditable, portableRequires recording/pushing tags
Docker ComposeRedeploy with IMAGE/TAG varsSimple, no extra toolingCompose file must use variables
## 9) Troubleshooting and hardening tips - Script stops without context: export TRACE=1 to enable set -x tracing from lib.sh. - Hidden failures in pipelines: always set -euo pipefail and quote variables ("${var}"). - Health check flapping: increase with_retry attempts or SLEEP; ensure your app returns 200 OK. - Kubectl not configured: provide KUBECONFIG or run kubectl config set-context before deploy. - Docker permission denied: ensure runner has access to Docker socket or use DinD with DOCKER_HOST. - Secrets in logs: never echo secret env vars; prefer files or masked CI variables. - Concurrent deploys: ensure lock_run uses a shared path (e.g., /tmp/deploy.lock) across steps. - Rollback fails on Compose: confirm compose.yaml uses image: "${IMAGE_REPO}:${TAG}" and that IMAGE_REPO/TAG are set from the recorded file. - Long-running commands: wrap with timebox 60 ; install coreutils/timeout if missing. ## 10) Local pilot you can run today - Make scripts executable: chmod +x scripts/*.sh - Optional .env for local runs: - APP_NAME=demoapp - IMAGE_REPO=example/demoapp - STAGE=staging - HEALTH_URL=http://localhost:8080/healthz - Run the flow: - scripts/lint.sh - scripts/test.sh - scripts/build.sh - scripts/validate.sh || true - scripts/deploy.sh - scripts/rollback.sh promote - Test rollback: - Deploy a bad tag (e.g., edit k8s/deployment.yaml or set TAG=bad and run deploy) - Observe failed health check - Run scripts/rollback.sh rollback and confirm recovery Track: total runtime, health-check success rate during deploy, and mean time to rollback. ## Conclusion You now have a practical Bash toolkit for CI/CD: strict shell safety, reusable logging and helpers, validation before deploy, verified rollouts, and fast, auditable rollbacks. Start with the local pilot, wire it into GitHub Actions or GitLab CI, and evolve the same scripts across staging and production. Rehearse failures regularly, keep previous versions recorded in durable storage, and your Bash automation will remain predictable, inspectable, and safe to extend as your platform grows.