## Intro Kubernetes Jobs are the workhorses for batch processing, one-off tasks, and maintenance operations. They ensure that a specified number of pods complete successfully, handling retries and pod lifecycle. But a Job is only as reliable as its configuration. A missing restartPolicy , an unbounded backoffLimit , or a missing resource limit can turn a simple task into a silent failure or a resource-hogging monster. In this guide, we will walk through the most common Kubernetes Job configuration mistakes, how to validate your Job manifests before applying them, and how to recover safely when things go wrong. Each section includes concrete YAML snippets, kubectl commands, and expected outputs so you can apply these practices immediately. ## Version and Environment Inventory Before you create or debug Jobs, you need to know your Kubernetes environment. This avoids surprises from deprecated APIs, resource constraints, or permission issues. ### 1. Check Kubernetes version Use kubectl version to see client and server versions. Jobs are stable in batch/v1 since Kubernetes 1.21, but features like podFailurePolicy and managedBy require newer versions. kubectl version --short Client Version: v1.27.3 Server Version: v1.27.3 If your server is older than 1.21, you must use batch/v1beta1 for Jobs, but that API is removed in 1.25. Plan an upgrade. ### 2. Check API deprecations As of Kubernetes 1.25, the batch/v1beta1 CronJob API is removed. Always use batch/v1 for both Jobs and CronJobs. You can check for warnings when applying resources by using kubectl apply --dry-run=client or by querying the API server with kubectl api-resources . ### 3. Verify resource quotas If your namespace has a ResourceQuota, it may limit the number of Jobs, pods, or CPU/memory. Check with: kubectl describe resourcequota -n dev Name: compute-quota Namespace: dev Resource Used Hard -------- ---- ---- requests.cpu 500m 2 requests.memory 1Gi 4Gi limits.cpu 1 4 limits.memory 2Gi 8Gi pods 3 10 jobs.batch 1 5 If you exceed these limits, your Job pods won't be scheduled. ### 4. Assess node capacity Ensure your cluster nodes have enough allocatable CPU and memory for the Job's pods plus overhead. Use: kubectl describe nodes | grep -A 5 "Allocated resources" Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1200m (60%) 0 (0%) memory 900Mi (45%) 0 (0%) ### 5. Confirm RBAC permissions You need at least create , get , list , watch , and delete on Jobs and pods, and get on logs. For minimal access, create a Role: apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: dev name: job-manager rules: - apiGroups: ["batch"] resources: ["jobs"] verbs: ["create", "get", "list", "watch", "delete", "update"] - apiGroups: [""] resources: ["pods", "pods/log"] verbs: ["get", "list", "watch"] Bind this role to your user or service account.
Quick check 1 of 2 Which restartPolicy values are appropriate for a Kubernetes Job? - Always - OnFailure or Never - Always or OnFailure - Never only
As discussed in the reference, Job is only appropriate for pods with RestartPolicy equal to OnFailure or Never. The default is Always, which is not suitable for Jobs.
## Safe Configuration Path A safe approach to deploying Jobs is to start with a minimal manifest, validate with a dry-run, apply to a development namespace, then add resource limits, timeouts, and retries. This prevents accidental blast radius. ### Step 1: Create a basic Job manifest Create a file job.yaml with a simple Job that prints a message and exits. apiVersion: batch/v1 kind: Job metadata: name: hello-job spec: template: spec: containers: - name: hello image: busybox:1.36 command: ["sh", "-c", "echo Hello Kubernetes! && sleep 5"] restartPolicy: Never backoffLimit: 4 Notice restartPolicy is set to Never ; this is required for Jobs (alternatively OnFailure ). The image tag is pinned to 1.36 , not latest . ### Step 2: Dry-run validation Use kubectl apply --dry-run=client to validate syntax and API field correctness without actually creating the Job. kubectl apply --dry-run=client -f job.yaml job.batch/hello-job created (dry run) For a more thorough validation that hits the API server without persisting, use --dry-run=server (Kubernetes 1.18+). This checks admission webhooks and quota. kubectl apply --dry-run=server -f job.yaml job.batch/hello-job created (server dry run) ### Step 3: Apply to a development namespace Create a dedicated namespace for testing, e.g., dev , and apply the manifest. kubectl create namespace dev kubectl apply -f job.yaml -n dev job.batch/hello-job created ### Step 4: Set resource requests and limits Never run a Job without resource requests and limits. This prevents a pod from consuming all node resources and ensures predictable scheduling. Add the following to the container spec: resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "200m" memory: "100Mi" Full Job snippet: apiVersion: batch/v1 kind: Job metadata: name: hello-job spec: template: spec: containers: - name: hello image: busybox:1.36 command: ["sh", "-c", "echo Hello Kubernetes! && sleep 5"] resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "200m" memory: "100Mi" restartPolicy: Never backoffLimit: 4 ### Step 5: Configure timeouts and retries Control how long a Job can run and how many times it retries failed pods. - backoffLimit : number of retries before marking the Job as failed. Default is 6. - activeDeadlineSeconds : maximum duration the Job can run, including all pod retries. Once exceeded, the Job is terminated and marked failed. - completions : how many successful pod completions are required for the Job to be considered complete. - parallelism : how many pods can run concurrently. Example: spec: backoffLimit: 2 activeDeadlineSeconds: 100 completions: 5 parallelism: 2 With this configuration, the Job creates 5 pods total, running 2 at a time. Each pod has up to 2 retries, and the entire Job must finish within 100 seconds. If activeDeadlineSeconds is exceeded, all pods are killed and the Job fails with reason DeadlineExceeded . ### Full safe Job manifest Combining all steps: apiVersion: batch/v1 kind: Job metadata: name: hello-job namespace: dev spec: backoffLimit: 2 activeDeadlineSeconds: 100 completions: 5 parallelism: 2 template: spec: containers: - name: hello image: busybox:1.36 command: ["sh", "-c", "echo Hello Kubernetes! && sleep 5"] resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "200m" memory: "100Mi" restartPolicy: Never ## Verification and Diagnostics After applying the Job, verify its status, inspect logs, and monitor events to ensure successful completion. ### Check Job status Use kubectl get jobs to see completion status. kubectl get jobs -n dev NAME COMPLETIONS DURATION AGE hello-job 5/5 45s 2m If COMPLETIONS shows 0/5 , the Job is still running or stuck. ### Describe the Job kubectl describe job provides details about pod statuses, events, and conditions. kubectl describe job hello-job -n dev Name: hello-job Namespace: dev Selector: controller-uid=1234abcd-... Labels: controller-uid=1234abcd-...,job-name=hello-job Annotations: Parallelism: 2 Completions: 5 Start Time: Mon, 12 Aug 2024 10:00:00 +0000 Pods Statuses: 2 Active / 3 Succeeded / 0 Failed Pod Template: Labels: controller-uid=1234abcd-...,job-name=hello-job Containers: hello: Image: busybox:1.36 Port: Command: ["sh", "-c", "echo Hello Kubernetes! && sleep 5"] Limits: cpu: 200m memory: 100Mi Requests: cpu: 100m memory: 50Mi Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 2m job-controller Created pod: hello-job-abcde Normal SuccessfulCreate 2m job-controller Created pod: hello-job-fghij Normal SuccessfulCreate 1m job-controller Created pod: hello-job-klmno Normal SuccessfulCreate 1m job-controller Created pod: hello-job-pqrst Normal SuccessfulCreate 1m job-controller Created pod: hello-job-uvwxy Look for FailedCreate events if pods cannot be created (e.g., quota exceeded). ### View Pod logs First, list pods associated with the Job using the job-name label selector. kubectl get pods -l job-name=hello-job -n dev NAME READY STATUS RESTARTS AGE hello-job-abcde 0/1 Completed 0 2m hello-job-fghij 0/1 Completed 0 2m hello-job-klmno 0/1 Completed 0 1m hello-job-pqrst 0/1 Completed 0 1m hello-job-uvwxy 0/1 Completed 0 1m Then view logs for any pod: kubectl logs hello-job-abcde -n dev Hello Kubernetes! For failed pods, logs often reveal the error. ### Monitor events Events provide a timeline of what happened to the Job and its pods. kubectl get events --field-selector involvedObject.name=hello-job-abcde -n dev LAST SEEN TYPE REASON OBJECT MESSAGE 2m Normal Scheduled pod/hello-job-abcde Successfully assigned dev/hello-job-abcde to node-1 2m Normal Pulled pod/hello-job-abcde Container image "busybox:1.36" already present on machine 2m Normal Created pod/hello-job-abcde Created container hello 2m Normal Started pod/hello-job-abcde Started container hello If a pod fails, look for Warning events such as BackOff , Failed , or Unhealthy .
Quick check 2 of 2 What does the Job controller do when it sees a new Job? - It runs the containers itself. - It tells the API server to create or remove Pods. - It directly schedules Pods on Nodes. - It updates the Job to mark it Finished immediately.
The Job controller does not run any Pods or containers itself. Instead, it tells the API server to create or remove Pods. Other components act on the new information to schedule and run the Pods.
## Failure Modes and Recovery Jobs can fail for many reasons. Here are common failure modes, symptoms, and recovery steps.
Failure ModeSymptomsRecovery
ImagePullBackOffPod stuck in Pending or ImagePullBackOff, events show Failed to pull imageVerify image name/tag, registry credentials, network access. Use kubectl describe pod for details.
CrashLoopBackOffPod restarts repeatedly, exit code non-zeroInspect logs (kubectl logs <pod>), fix command or environment variables.
DeadlineExceededJob status shows Failed with reason DeadlineExceededIncrease activeDeadlineSeconds or optimize workload.
ResourceQuotaExceededJob cannot create pods, events show exceeded quotaAdjust resource requests or increase namespace quota.
EvictedPods evicted due to node pressureIncrease node resources or reduce pod resource requests.
BackoffLimitExceededJob failed after backoffLimit retriesInvestigate root cause, increase backoffLimit if transient.
### Detailed recovery examples ### ImagePullBackOff Suppose your Job uses image: myapp:latest but the registry requires authentication. The pod will show ImagePullBackOff . Check the pod: kubectl describe pod myjob-xxxxx -n dev Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 5m default-scheduler Successfully assigned dev/myjob-xxxxx to node-1 Warning Failed 5m kubelet Failed to pull image "myapp:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for myapp, repository does not exist or may require 'docker login' Warning Failed 5m kubelet Error: ErrImagePull Normal BackOff 4m kubelet Back-off pulling image "myapp:latest" Warning Failed 4m kubelet Error: ImagePullBackOff Recovery: - Verify the image name and tag, e.g., myregistry.com/myapp:1.2.3 . - If private, create a Secret with registry credentials and add imagePullSecrets to the pod spec. - Reapply the corrected manifest. ### CrashLoopBackOff A pod that exits with a non-zero code repeatedly will enter CrashLoopBackOff . kubectl get pods -n dev NAME READY STATUS RESTARTS AGE myjob-xxxxx 0/1 CrashLoopBackOff 5 3m Inspect logs: kubectl logs myjob-xxxxx -n dev --previous Error: cannot open config file Recovery: - Fix the command or environment to provide the missing config. - Update the Job manifest and reapply. - If the Job is still running, delete the pod to trigger a new one (subject to backoffLimit). ### DeadlineExceeded If your Job exceeds activeDeadlineSeconds , all pods are terminated and the Job fails. kubectl get job myjob -n dev NAME COMPLETIONS DURATION AGE myjob 0/1 100s 2m kubectl describe job myjob -n dev ... Conditions: Type Status Reason Message ---- ------ ------ ------- Failed True DeadlineExceeded Job was active longer than specified deadline Recovery: - If the workload legitimately needs more time, increase activeDeadlineSeconds . - If it hung, investigate the root cause via logs. - Delete the failed Job and create a new one with corrected settings. ### Rollback and recovery steps When a Job fails due to misconfiguration, follow these steps: - Edit the Job manifest to fix the issue. If you have the original file, modify it; otherwise, use kubectl get job -o yaml > job.yaml to retrieve it. - Reapply the manifest with kubectl apply . Note that some Job fields are immutable (e.g., template , backoffLimit , completions , parallelism after creation in older versions). If you need to change immutable fields, you must delete and recreate the Job. - If immutable fields changed , delete the old Job and create a new one: kubectl delete job hello-job -n dev kubectl apply -f corrected-job.yaml -n dev - Clean up completed Jobs to avoid clutter and resource leaks. Use kubectl delete job --all -n dev for all Jobs in the namespace, or set ttlSecondsAfterFinished on the Job to auto-delete after a period (Kubernetes 1.21+). spec: ttlSecondsAfterFinished: 3600 # delete 1 hour after completion ## Operations Checklist Before promoting a Job to production, ensure these items are in place. This checklist covers the most critical misconfiguration points. - Container image tag is pinned (e.g., busybox:1.36 , not latest ). - restartPolicy is set to Never or OnFailure (never Always ). - Resource requests and limits are defined for every container. - backoffLimit and activeDeadlineSeconds are set based on expected runtime and retry tolerance. - Manifest validated with kubectl apply --dry-run=server in a development namespace. - Logs are accessible via your logging stack and monitored for errors. - Rollback plan documented and tested (e.g., you can redeploy from version control). - Job is idempotent or safe to re-run (no side effects if run twice). - completions and parallelism match the workload (not accidentally left at defaults that over- or under-provision). - Namespace quotas and node capacity are sufficient for peak parallelism. - RBAC permissions are minimal but sufficient for automation. - Job has appropriate labels for identification and monitoring (e.g., app: billing-export ). Use this checklist as a gate in your CI/CD pipeline or as a review document for manual deployments. ## Conclusion Kubernetes Jobs are powerful, but misconfigurations are common and can lead to silent failures, infinite retries, or resource exhaustion. By following a safe configuration path, validating with dry-runs, setting resource limits and timeouts, and monitoring events, you can run Jobs reliably in production. Remember to start with a narrow pilot, validate locally, and establish monitoring and rollback habits. Regularly review your Job configurations to prevent drift. With these practices, your batch workloads will run smoothly, and when they fail, you will know exactly how to diagnose and recover.