Bash troubleshooting with practical examples: practical implementation guide

Intro

Bash is the glue that holds many build, release, and ops tasks together. When it fails, the blast radius can be large. This guide is a hands-on reference to diagnose and fix Bash problems with practical, copy-paste snippets. You will learn a simple workflow, which logs to capture, safe commands that prevent data loss, and step-by-step recovery patterns you can run locally, in Docker and Docker Compose, and inside Kubernetes pods.

Workflow Overview

Use this repeatable flow from symptom to validated fix:

1. Capture the symptom

• Record the exact command, arguments, working directory, and exit code: echo $?.
• Save stderr and stdout separately: cmd 1>out.log 2>err.log.
• Record versions: bash --version, printf '%s\n' "$BASH_VERSION".

1. Minimize the failure

• Reduce the script to the smallest snippet that still fails.
• Replace external dependencies with fakes or fixtures.

1. Trace and log

• Enable tracing only around the suspect lines:

  {
    set -x
    # suspect block
    do_thing
    set +x
  } 2>trace.log

• Add timestamps to traces for slow steps:

  export PS4='+ ${EPOCHREALTIME} ${BASH_SOURCE}:${LINENO}: '

1. Run root-cause checks

• Verify PATH, quoting, permissions, line endings, and pipeline behavior.

1. Apply a safe fix

• Use non-destructive flags, dry runs, and temp files.

1. Prove and document

• Re-run the minimized repro and the original script.
• Add a test case or example so the issue does not recur.

Collecting Logs

Good logs answer: what failed, where, and why.

• Basic capture

  # Exit code and stderr
  failing_cmd 1>stdout.log 2>stderr.log; echo $? >exit.code

  # Full trace to a separate fd to avoid mixing with program output
  exec {XTRACE_FD}>trace.log
  export BASH_XTRACEFD=$XTRACE_FD
  export PS4='+ ${EPOCHREALTIME} ${BASH_SOURCE}:${LINENO}: '
  set -x
  failing_cmd
  set +x
  exec {XTRACE_FD}>&-

• Keep secrets out of logs

  set +x  # disable tracing around secret material
  token="${TOKEN:?missing TOKEN}"
  set -x

• Container and cluster logs
• Docker: docker logs <container> 2>&1 | tee container.log
• Compose: docker compose logs svc -f
• Kubernetes: kubectl logs pod/my-pod -c my-container --timestamps

• Environment snapshots (avoid secrets)

  ( env | grep -Ev 'KEY|SECRET|TOKEN|PASS' ) | sort >env.snapshot

Root-cause Checks

Target common failure classes before you guess.

1. PATH and command resolution

which mycmd || type -a mycmd || echo 'missing'
hash -r  # clear Bash command hash if binaries moved
printf '%s\n' "$PATH" | tr ':' '\n'

1. Shebang and shell differences

head -1 script.sh  # should be: #!/usr/bin/env bash
bash -n script.sh  # syntax check
# Some distros run /bin/sh -> dash; force bash when needed
bash script.sh

1. Permissions and executables

chmod +x script.sh
file script.sh  # check for CRLF
# If CRLF found, normalize:
dos2unix script.sh  # or: sed -i 's/\r$//' script.sh

1. Quoting and globbing

# Safe var expansion
echo "$var"
# Disable globbing when expanding user input
set -f; printf '%s\n' $user_input; set +f
# Safer: always quote
printf '%s\n' "$user_input"

1. Pipelines and exit codes

set -o pipefail  # fail if any command in the pipeline fails
# Capture pipeline status
foo | bar | baz; ec=$?; echo "pipeline exit=$ec"
# If a non-match with grep is acceptable:
grep -q pattern file || true

1. Unbound variables and defaults

set -u   # treat unset variables as errors
# Provide defaults or hard errors
name="${NAME:-anonymous}"
api="${API_URL:?API_URL is required}"

1. stdin-safe loops

# Process lines with preserved whitespace and no globbing
while IFS= read -r line; do
  printf 'line: %s\n' "$line"
done < input.txt

1. Temporary files and races

f=$(mktemp) || exit 1
trap 'rm -f "$f"' EXIT
# use "$f" safely here

Safe Commands

Prefer non-destructive operations during recovery.

• Dry runs and no-clobber

rsync -av --dry-run src/ dst/
cp -an src/ dst/   # -n no-clobber, -a preserve
mv -i old new      # -i interactive prompt
rm -i file         # confirm before delete

• Backups and checkpoints

cp -a dir "dir.bak.$(date +%Y%m%d-%H%M%S)"

• Guardrails

set -euo pipefail  # careful: changes failure semantics
# Narrow the scope to a block
do_safe() {
  set -euo pipefail
  "$@"
}

• Rate limiting and timeouts

timeout 10s long_running || echo 'timed out'

Step-by-step Recovery

Below are practical playbooks you can adapt.

1) Command not found or wrong binary

Symptoms: command not found or an unexpected version runs.

Checks

type -a tool
printf '%s\n' "$PATH" | tr ':' '\n'
hash -r

Fixes

• Install or adjust PATH in the script: export PATH=/opt/tool/bin:$PATH.
• Use absolute paths for critical binaries.
• In containers, bake the dependency into the image or install at start.

2) Unbound variable crash

Symptoms: unbound variable with set -u.

Checks

set -u
: "${NAME:?NAME is required}"

Fixes

• Provide defaults or hard fails where appropriate:

  NAME="${NAME:-guest}"
  API_URL="${API_URL:?required}"

3) Pipeline hides failure

Symptoms: Last command succeeds but an early stage failed.

Checks and fix

set -o pipefail
foo | bar | baz

If a non-zero from grep on no-match is acceptable, append || true.

4) Permission or shebang error

Symptoms: Permission denied or script runs under sh instead of Bash.

Checks and fixes

chmod +x script.sh
sed -n '1p' script.sh  # expect #!/usr/bin/env bash
bash -n script.sh

5) CRLF line endings break execution

Symptoms: ^M appears or scripts fail in Linux containers.

Fix

file script.sh
dos2unix script.sh  # or sed -i 's/\r$//' script.sh

6) Quoting bug expands unexpectedly

Symptoms: Filenames with spaces break loops or rm targets.

Fix pattern

# Always quote expansions
rm -- "$file"
# Safe find + xargs
find . -type f -name '*.log' -print0 | xargs -0 rm -f --

7) Cron vs interactive differences

Symptoms: Script works in shell, fails under cron or a job runner.

Checks

# Cron has a minimal environment
( env | sort ) >interactive.env
( env -i bash -lc 'env' | sort ) >isolated.env
diff -u interactive.env isolated.env || true

Fixes

• Set PATH explicitly in the script.
• Use absolute paths for files and binaries.
• Source a known profile file if required, or export needed vars in the crontab entry.

Docker and Docker Compose specifics

• Reproduce inside the same image

  docker run --rm -it -v "$PWD":/work -w /work myimg bash -lc './script.sh'
  docker exec -it mycontainer bash -lc 'type -a tool && ./script.sh'

• Check file ownership and permissions after bind mounts.
• Normalize line endings on the host before building images.
• Compose service shell

  docker compose run --rm svc bash -lc 'set -euxo pipefail; ./script.sh'

Kubernetes specifics (Pods, Ingress, Secrets)

• Exec into the container with the right shell

  kubectl exec -it deploy/app -c app -- bash -lc 'type -a bash || cat /etc/os-release'

• Logs and restarts

  kubectl logs deploy/app -c app --timestamps --previous

• Config and secrets

  # Inspect mounted files without printing values to the screen history
  ls -l /etc/config /etc/secret
  stat /etc/secret/*
  # Avoid tracing secret expansions
  set +x; KEY="${API_KEY:?missing}"; set -x

• Network checks relevant to Ingress and services

  curl -sv http://svc.namespace.svc.cluster.local:8080/health || true
  getent hosts example.com
  nc -zv example.com 443 || true

Local Pilot Plan

Keep your first implementation small, measurable, and easy to inspect locally.

1. Pick 3 failures you see often

• Example set: unbound variable, CRLF lines, and hidden pipeline failure.

1. Create minimal repros

# repro-unbound.sh
set -u; echo "Hello $NAME"

# repro-crlf.sh (intentionally CRLF)
printf 'echo win\r\n' > repro-crlf.sh

# repro-pipe.sh
false | grep x

1. Add a trace-and-log wrapper

run_traced() {
  exec {XTRACE_FD}>trace.log
  export BASH_XTRACEFD=$XTRACE_FD
  export PS4='+ ${EPOCHREALTIME} ${BASH_SOURCE}:${LINENO}: '
  set -x; "$@"; ec=$?; set +x
  exec {XTRACE_FD}>&-
  return "$ec"
}
run_traced bash repro-unbound.sh || true

1. Measure time-to-fix

• Capture how long it takes to identify and fix before and after adding the wrapper and checks.
• A simple timer: time bash repro-unbound.sh.

1. Expand to containers and clusters

• After the local pilot works, reproduce inside your Docker image and a Kubernetes pod, then apply the same checks and wrappers.

Conclusion

Bash troubleshooting gets faster when you follow a clear workflow, capture the right logs, use safe recovery commands, and apply proven playbooks for common failures. Start with a small local pilot, measure the impact, then extend the same patterns to Docker, Docker Compose, and Kubernetes. Keep growing your examples and snippets so future incidents are predictable and low risk.