## Intro

Bash still powers build, release, and operations flows across servers, CI, and containers. When a shell script fails, the blast radius can be wide. This guide is a practical, copy-paste reference to diagnose and fix Bash problems quickly and safely. You will learn a clear workflow, the logs that matter, safe commands that avoid data loss, and step-by-step playbooks you can run locally, in Docker or Compose, and inside Kubernetes pods.

## Workflow overview

Use this repeatable flow from symptom to verified fix:

- Capture the symptom

- Record the exact command, arguments, working directory, and exit code:

pwd; printf 'exit=%s\n' "$?" 

- Save stdout and stderr separately:

 cmd 1>out.log 2>err.log; printf '%s\n' "$?" >exit.code 

- Record Bash version and platform:

 bash --version
 printf '%s\n' "$BASH_VERSION"
 uname -a 

- Minimize the failure

- Reduce to the smallest snippet that still fails.

- Replace external calls with stubs or fixtures; pin inputs.

- Trace and log

- Enable tracing only around suspect lines and send it to a separate file:

 exec {XTRACE_FD}>trace.log
 export BASH_XTRACEFD=$XTRACE_FD
 export PS4='+ ${EPOCHREALTIME:-$EPOCHSECONDS} ${BASH_SOURCE}:${LINENO} ${FUNCNAME[0]:-main}: '
 set -x
 # suspect block
 do_thing
 set +x
 exec {XTRACE_FD}>&- 

- Run root-cause checks

- Verify PATH, command resolution, shebang, permissions, quoting, line endings, and pipeline behavior.

- Apply a safe fix

- Prefer dry runs, no-clobber flags, and temporary files.

- Prove and document

- Re-run both the minimized repro and the original script.

- Add a test or example so the issue does not recur.

## Collecting logs that answer what, where, and why

 Basic capture

# Exit code, stdout, stderr
failing_cmd 1>stdout.log 2>stderr.log; printf '%s\n' "$?" >exit.code

# Xtrace with timestamps and file:line:function, without polluting stdout/stderr
exec {XTRACE_FD}>trace.log
export BASH_XTRACEFD=$XTRACE_FD
export PS4='+ ${EPOCHREALTIME:-$EPOCHSECONDS} ${BASH_SOURCE}:${LINENO} ${FUNCNAME[0]:-main}: '
set -x
failing_cmd || true
set +x
exec {XTRACE_FD}>&- 
 Keep secrets out of logs

set +x
token="${TOKEN:?missing TOKEN}"
set -x 
 Container and cluster logs

# Docker
docker logs my-container 2>&1 | tee container.log

# Docker Compose
docker compose logs my-service -f

# Kubernetes
kubectl logs pod/my-pod -c my-container --timestamps 
 Environment snapshot (filter obvious secrets)

(env | grep -Ev 'KEY|SECRET|TOKEN|PASS' ) | sort > env.snapshot 
 Tip: If Bash < 5 (no EPOCHREALTIME), PS4 will fall back to EPOCHSECONDS in the snippet above.

## Root-cause checks (quick wins first)

PATH and command resolution

command -v mycmd || { type -a mycmd || printf 'missing\n'; }
hash -r # clear Bash command hash if binaries moved
printf '%s\n' "$PATH" | tr ':' '\n' 
 Shebang and shell differences

head -1 script.sh # expect: #!/usr/bin/env bash
bash -n script.sh # syntax check (no execution)
# Force bash if /bin/sh maps to dash or ash
bash script.sh 
 Permissions and executables

chmod +x script.sh
file script.sh # detect CRLF
# Normalize line endings if needed
sed -i 's/\r$//' script.sh # or: dos2unix script.sh 
 Quoting, word splitting, and globbing

# Always quote expansions that may contain spaces/newlines
printf '%s\n' "$var"

# Prefer arrays for safe token lists
files=("*.log") # beware: expands immediately in current shell
# Safer: use find -print0 / xargs -0 for filenames
find . -type f -name '*.log' -print0 | xargs -0 printf '%s\n'

# If you must expand user input, disable globbing temporarily
set -f; printf '%s\n' $user_input; set +f 
 Pipelines and exit codes

set -o pipefail # fail if any command in the pipeline fails
foo | bar | baz
printf 'pipeline exit=%s\n' "$?"
# Inspect each stage’s exit code precisely
foo | bar | baz; printf '%s\n' "${PIPESTATUS[@]}"
# If a non-match from grep is acceptable:
grep -q pattern file || true 
 Unbound variables and defaults

set -u # treat unset variables as errors
name="${NAME:-anonymous}"
api="${API_URL:?API_URL is required}" 
 stdin-safe loops

# Preserve whitespace, backslashes, and do not glob
while IFS= read -r line; do
 printf 'line: %s\n' "$line"
done < input.txt

# Reading a file into an array safely
declare -a lines=()
mapfile -t lines < input.txt # aka readarray 
 Temporary files and races

f=$(mktemp) || exit 1
trap 'rm -f "$f"' EXIT
# use "$f" safely here 
 Static analysis

# Fast feedback for quoting, test conditions, arrays, etc.
shellcheck script.sh || true 

## Safe commands during incident recovery

 Dry runs and no-clobber

rsync -av --dry-run src/ dst/
cp -an src/ dst/ # -n no-clobber, -a preserve attrs
mv -i old new # -i interactive prompt
rm -i file # confirm before delete 
 Backups and checkpoints

cp -a dir "dir.bak.$(date +%Y%m%d-%H%M%S)" 
 Guardrails (use consciously)

# Scope strict mode to a function to avoid surprising control-flow changes
do_safe() {
 set -Eeuo pipefail
 shopt -s inherit_errexit 2>/dev/null || true # Bash >=4.4
 trap 'printf "ERR at %s:%s in %s\n" "${BASH_SOURCE[0]}" "$LINENO" "${FUNCNAME[0]:-main}" >&2' ERR
 "$@"
} 
 Timeouts and rate limiting

timeout 10s long_running || printf 'timed out\n' 
 Note: set -e changes failure semantics in conditionals, subshells, and pipelines. Prefer explicit checks, pipefail, and targeted do_safe blocks.

## Step-by-step recovery playbooks

- Command not found or wrong binary

Symptoms: command not found, or an unexpected version runs.

Checks

type -a tool || true
printf '%s\n' "$PATH" | tr ':' '\n'
hash -r 
 Fixes

- Install the tool or adjust PATH in the script:

export PATH=/opt/tool/bin:$PATH 

- Use absolute paths for critical binaries (e.g., /usr/bin/env bash, /usr/bin/grep).

- In containers, bake dependencies into the image or install them at container start.

- On Alpine, /bin/sh is busybox ash; prefer bash explicitly if your script needs it.

- Unbound variable crash

 Symptoms: script exits with unbound variable (often paired with set -u).

Checks

set -u
: "${NAME:?NAME is required}" 
 Fixes

NAME="${NAME:-guest}"
API_URL="${API_URL:?required}" 

- Pipeline hides a failure

 Symptoms: the last command succeeds but an early stage failed.

Checks and fix

set -o pipefail
foo | bar | baz
printf '%s\n' "${PIPESTATUS[@]}"
# If grep no-match should not fail the pipeline:
grep -q pattern file || true 

- Permission or shebang error

 Symptoms: Permission denied or script runs under sh instead of Bash.

Checks and fixes

chmod +x script.sh
sed -n '1p' script.sh # expect #!/usr/bin/env bash
bash -n script.sh 

- CRLF line endings break execution

 Symptoms: ^M appears or scripts fail in Linux containers.

Fix

file script.sh
sed -i 's/\r$//' script.sh # or: dos2unix script.sh 

- Quoting bug expands unexpectedly

 Symptoms: filenames with spaces or wildcards break loops or rm targets.

Fix pattern

# Always quote expansions and pass separators first
rm -- "$file"
# Safe find + xargs
find . -type f -name '*.log' -print0 | xargs -0 rm -f -- 

- Cron vs interactive differences

 Symptoms: script works in an interactive shell, fails under cron or a job runner.

Checks

# Cron has a minimal environment
(env | sort) > interactive.env
(env -i bash -lc 'env' | sort) > isolated.env
diff -u interactive.env isolated.env || true 
 Fixes

- Export PATH explicitly at the top of the script.

- Use absolute paths for files and binaries.

- Source a minimal, known profile (or export required vars) from the crontab entry.

## Docker and Docker Compose specifics

- Reproduce inside the same image:

docker run --rm -it -v "$PWD":/work -w /work myimg bash -lc './script.sh'
 docker exec -it mycontainer bash -lc 'type -a tool && ./script.sh' 

- Check file ownership and permissions after bind mounts (host UID/GID mismatches are common).

- Normalize line endings on the host before building images.

- Compose service shell with strict mode for fast failure:

 docker compose run --rm svc bash -lc 'set -Eeuo pipefail; ./script.sh' 

## Kubernetes specifics (pods, restarts, secrets)

- Exec into the container with the right shell:

 kubectl exec -it deploy/app -c app -- bash -lc 'type -a bash || cat /etc/os-release' 

- Inspect logs, including previous container if it crashed:

 kubectl logs deploy/app -c app --timestamps --previous 

- Inspect mounted config and secret paths without printing secret values:

 ls -l /etc/config /etc/secret || true
 stat /etc/secret/* 2>/dev/null || true
 set +x; KEY="${API_KEY:?missing}"; set -x 

- Network checks relevant to services and Ingress:

 curl -sv http://svc.namespace.svc.cluster.local:8080/health || true
 getent hosts example.com || true
 nc -zv example.com 443 || true 

- If the target container lacks Bash, consider an ephemeral debug container:

 # Requires cluster support and permissions
 kubectl debug -it deploy/app --image=bash:5 --target=app -- bash 

## Local pilot plan

 Keep your first implementation small, measurable, and easy to inspect locally.

- Pick 3 failures you see often

- Example set: unbound variable, CRLF lines, hidden pipeline failure.

- Create minimal repros

# repro-unbound.sh
set -u; echo "Hello $NAME"

# repro-crlf.sh (intentionally CRLF)
printf 'echo win\r\n' > repro-crlf.sh

# repro-pipe.sh
false | grep x 

- Add a trace-and-log wrapper

 run_traced() {
 exec {XTRACE_FD}>trace.log
 export BASH_XTRACEFD=$XTRACE_FD
 export PS4='+ ${EPOCHREALTIME:-$EPOCHSECONDS} ${BASH_SOURCE}:${LINENO} ${FUNCNAME[0]:-main}: '
 set -x; "$@"; ec=$?; set +x
 exec {XTRACE_FD}>&-
 return "$ec"
}
run_traced bash repro-unbound.sh || true 

- Measure time to fix

- Compare before vs after adding the wrapper and checks.

- Simple timer:

 time bash repro-unbound.sh || true 

- Expand to containers and clusters

- Reproduce inside your Docker image and a Kubernetes pod; apply the same checks and wrapper.

## Conclusion

 Bash troubleshooting gets faster when you follow a clear workflow, capture high-signal logs, apply safe recovery commands, and use proven playbooks for common failures. Start locally with a small pilot, measure the impact, then extend the same checks and wrappers to Docker, Docker Compose, and Kubernetes. Keep growing your examples and snippets so future incidents become predictable and low risk.