Kubernetes Jobs and CronJobs troubleshooting with practical examples

Kubernetes Jobs run short-lived workloads to completion; CronJobs schedule those Jobs on a recurring timetable. When either fails, you need a repeatable way to find the root cause fast. This guide shows how retries and backoffLimit work, where to look for failed Pods, how to read logs and events, and how to troubleshoot safely in production-like clusters. You will also get copy-paste manifests and commands you can adapt today.

Workflow Overview

Use this path to diagnose most Job and CronJob issues:

Observe the controller

kubectl get job, kubectl get cronjob -o wide
Note status fields (succeeded, failed, active, last schedule, suspend, concurrencyPolicy)

Find the Pods

For a Job: kubectl get pods -l job-name=<job>
For a CronJob run: list Jobs with ownerReferences or name pattern, then list their Pods

Describe for events and reasons

kubectl describe job/<name>
kubectl describe pod/<pod> (look for Reason/Message like ImagePullBackOff, OOMKilled, Error, FailedScheduling)

Read logs

kubectl logs <pod> [-c <container>]
If the container crashed, add --previous
For all Pods of a Job: kubectl logs -l job-name=<job> --all-containers=true

Check spec pitfalls

Jobs: backoffLimit, activeDeadlineSeconds, parallelism, completions, restartPolicy
CronJobs: schedule, startingDeadlineSeconds, concurrencyPolicy, suspend, history limits

Act safely

Pause CronJob with: kubectl patch cronjob/<name> -p '{"spec":{"suspend":true}}'
Reproduce by creating an ad-hoc Job from a CronJob: kubectl create job --from=cronjob/<cron> <tmp-job>
Roll forward with a small change, watch events and logs, then re-enable

Jobs vs CronJobs in practice

Jobs orchestrate Pods until the desired number of completions succeed. CronJobs create Jobs on a schedule. The key troubleshooting differences:

Jobs: focus on Pod lifecycle, exit codes, backoffLimit, and resource issues.
CronJobs: add schedule evaluation, concurrencyPolicy, missed runs (startingDeadlineSeconds), and history cleanup. You often debug the spawned Job first, then come back to the CronJob spec if scheduling is the problem.

Troubleshooting a failing Job

Confirm state

kubectl get job <name> -o wide

Look at Succeeded, Failed, Active.

Check Pods

kubectl get pods -l job-name=<name>

If none exist: check FailedScheduling or quota issues with kubectl describe job and kubectl describe events.

Inspect events and status

kubectl describe pod <pod>

Common signals:

ImagePullBackOff: bad image or missing pull secret.
ErrImagePull: registry or auth error.
OOMKilled: memory limit too low for the process.
Error/Completed: check exitCode in container status.

Read logs

kubectl logs <pod> --all-containers=true

If the container restarts within the same Pod (restartPolicy: OnFailure), look at --previous.

Understand retries and backoffLimit

backoffLimit: maximum Pod retries before the Job is marked Failed. Example: backoffLimit: 2 allows two Pod retries after the first failure (3 attempts total).
activeDeadlineSeconds: hard time limit for the Job; exceeding it fails the Job.

Fix patterns

Image errors: correct the image tag or add imagePullSecrets.
Resource pressure: raise memory/cpu limits or optimize the workload.
Process exit code non-zero: handle errors in the app, add retries, or adjust inputs.
Slow jobs: extend activeDeadlineSeconds or reduce parallelism to ease pressure.

Troubleshooting a failing CronJob

List CronJobs

kubectl get cronjob -o wide

Check SCHEDULE, SUSPEND, ACTIVE, LAST SCHEDULE.

Is it creating Jobs?

If ACTIVE stays 0 and LAST SCHEDULE does not change, the controller may be blocked or schedule filters apply. Inspect:

suspend: true means no runs will be created.
startingDeadlineSeconds: missed schedules older than this are skipped.
concurrencyPolicy: Forbid prevents a new run if the prior run is still active; Replace cancels the active run.

Inspect the latest spawned Job

kubectl get jobs --sort-by=.metadata.creationTimestamp

Take the newest Job owned by the CronJob and debug it like any other Job (events, Pods, logs).

Safe operations

Temporarily pause: kubectl patch cronjob/<name> -p '{"spec":{"suspend":true}}'
Test changes by creating a one-off Job from the template: kubectl create job --from=cronjob/<name> <tmp-job>
Resume after verifying: kubectl patch cronjob/<name> -p '{"spec":{"suspend":false}}'

Practical examples

Example 1: A Job that fails twice then succeeds

apiVersion: batch/v1
kind: Job
metadata:
  name: demo-backoff
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: worker
        image: bash:5
        command: ["bash", "-lc", "if [ ! -f /tmp/ok ]; then echo first or second fail; touch /tmp/ok; exit 1; else echo success; fi"]

What to observe:

First Pod exits 1; controller retries up to backoffLimit.
Second attempt also exits 1; third attempt runs and prints success.
kubectl describe job/demo-backoff shows Failed increments before Succeeded.

Example 2: CronJob blocked by concurrencyPolicy=Forbid

apiVersion: batch/v1
kind: CronJob
metadata:
  name: report-job
spec:
  schedule: "*/5 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 2
  failedJobsHistoryLimit: 2
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: reporter
            image: bash:5
            command: ["bash", "-lc", "echo starting; sleep 400; echo done"]

Why it fails to schedule every 5 min:

Each run takes ~6.6 min (400s). With Forbid, the 5-min trigger that arrives while a run is active will be skipped. LAST SCHEDULE may not advance every tick.
Fix options: set concurrencyPolicy: Allow, reduce runtime, or increase interval.

Example 3: ImagePullBackOff in a Job

apiVersion: batch/v1
kind: Job
metadata:
  name: bad-image
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: c
        image: example.com/private/app: does-not-exist

Troubleshoot:

kubectl describe pod shows ErrImagePull/ImagePullBackOff.
Fix the tag or configure imagePullSecrets and correct registry URL.

Example 4: OOMKilled in a Job

apiVersion: batch/v1
kind: Job
metadata:
  name: mem-tight
spec:
  template:
    spec:
      restartPolicy: OnFailure
      containers:
      - name: c
        image: python:3.11
        resources:
          requests: { memory: "64Mi", cpu: "100m" }
          limits:   { memory: "64Mi", cpu: "200m" }
        command: ["python", "-c", "a='x'*200_000_000; print(len(a))"]

Troubleshoot:

kubectl describe pod shows OOMKilled in last state.
Raise memory limit (for example 256Mi) or reduce memory use in the program.

Local Pilot Plan

Start small and measurable:

Pick one Job with clear success criteria

Example: transform a small input file and write a checksum.
Define done as exit code 0 and presence of an output line.

Make it easy to inspect

Add structured logs and a final success message.
Set backoffLimit to 1 and activeDeadlineSeconds to a safe upper bound.

Validate in a local cluster

Apply the Job, watch events, and read logs end-to-end.
Record the run time and resource use.

Wrap it in a CronJob (optional)

Choose a conservative schedule first (for example every hour).
Set concurrencyPolicy to Forbid until you prove idempotency.

Expand gradually

Increase input size, then parallelism.
Add alerts on Job failures and missed schedules.

This approach reduces risk and shortens the feedback loop.

Conclusion

Jobs and CronJobs fail for understandable reasons: bad images, resource pressure, non-zero exits, or scheduling rules. The fastest path is consistent: check controller status, find the Pods, read events, inspect logs, and validate spec fields such as backoffLimit, activeDeadlineSeconds, schedule, and concurrencyPolicy. Use suspend and ad-hoc Jobs to test safely, then roll forward in small steps. Keep runs short, outputs clear, and success criteria obvious. That is how you troubleshoot with confidence.