## Intro Kubernetes CronJobs are the go-to mechanism for running time-based workloads such as database backups, report generation, cache warming, or clean-up tasks. A CronJob behaves much like the classic Unix cron daemon, but instead of executing a process on a single host, it creates Kubernetes Jobs on a repeating schedule. Each Job, in turn, creates one or more Pods that perform the actual work. This layering gives you scheduling flexibility plus Kubernetes-native retries, parallelism, and resource management. For operators, the real challenge is not writing the CronJob YAML; it is knowing which commands to run when a schedule has drifted, a Job has failed silently, a Pod is crash-looping, or a resource quota is being exhausted. This guide focuses on the practical command set you need to inspect, configure, debug, and recover CronJobs. Every section includes concrete commands, expected output, and failure signals, so you can move from observation to resolution without guesswork. Throughout, we emphasize operational safety: observe before changing, limit the blast radius of any modification, use placeholders instead of secrets in tests, verify the result, and document recovery paths before you need them. Whether you are a developer responsible for a single microservice or a DevOps consultant managing a shared cluster, these commands will help you keep scheduled workloads healthy and predictable. ## Version and Environment Inventory Before you can reliably operate CronJobs, you must know the Kubernetes version you are targeting and the API version of the CronJob resource. CronJobs have been stable (apiVersion: batch/v1) since Kubernetes 1.21. If you are on an older cluster, you may need batch/v1beta1, which was deprecated in 1.21 and removed in 1.25. The command to check your cluster version is: ```bash kubectl version --short ``` Expected output includes both client and server versions, such as: ``` Client Version: v1.28.2 Server Version: v1.27.3 ``` If your server version is below 1.21, you must use batch/v1beta1 in your CronJob manifests, and some features like the timeZone field (introduced in 1.24) will not be available. Next, confirm that the CronJob controller is running. The controller is part of the kube-controller-manager, which typically runs as a static Pod on control-plane nodes. To see it: ```bash kubectl get pods -n kube-system | grep controller-manager ``` You should see a Pod named something like kube-controller-manager- with status Running. If it is not running, CronJob scheduling will not work, and no Jobs will be created. Prerequisites for using CronJobs: - A running Kubernetes cluster with sufficient RBAC permissions to create, list, and describe CronJobs in your namespace. Use `kubectl auth can-i create cronjobs --namespace ` to verify. - A container image that is executable and exits with status 0 on success. Images that run forever will cause Jobs to hang until activeDeadlineSeconds is set. - Adequate resource quotas. A CronJob that requests more CPU or memory than the namespace allows will fail to schedule its Pods. A good starting observation is to list all CronJobs in your current namespace and their basic state: ```bash kubectl get cronjobs ``` Example output: ``` NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE backup-db 0 2 * * * False 0 2h 3d report-gen */15 * * * * False 1 15m 10d ``` For a deeper look at a specific CronJob, use describe: ```bash kubectl describe cronjob backup-db ``` This shows the schedule, concurrency policy, last successful run, and any recent events, such as failed to create job due to quota exceeded. Before making any change, capture the current state with a timestamp: ```bash date -u && kubectl get cronjob backup-db -o yaml > backup-db-before.yaml ``` Store this snapshot in a safe location. If the subsequent change causes problems, you can restore with `kubectl apply -f backup-db-before.yaml`. Avoid copying secrets or credentials into these snapshots; if they appear, redact them before saving. ## Safe Configuration Path CronJob configuration changes should follow a clear path: observe the current configuration, test a copy locally, apply the smallest diff, verify, and document rollback. Never edit a live CronJob directly with `kubectl edit` unless it is a genuine emergency; use a versioned manifest in source control instead. Here is a minimal safe CronJob manifest to use as a starting point: ```yaml apiVersion: batch/v1 kind: CronJob metadata: name: hello-world namespace: default spec: schedule: "*/5 * * * *" concurrencyPolicy: Forbid successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 1 jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - name: hello image: busybox:1.36 command: ["/bin/sh", "-c", "echo Hello Kubernetes; sleep 10"] resources: requests: cpu: "10m" memory: "20Mi" limits: cpu: "100m" memory: "50Mi" ``` Key fields and their safe values: - schedule: Use standard cron syntax with five fields. Be mindful of time zones; from Kubernetes 1.24 you can set `timeZone: "America/New_York"` to avoid confusion when nodes run in UTC. - concurrencyPolicy: Always set this explicitly. Forbid prevents overlapping runs if a Job takes longer than the interval. Replace means a new Job replaces a running one, and Allow permits concurrent runs. Choose based on workload safety; Forbid is usually the safe default for idempotent jobs. - startingDeadlineSeconds: If set, the CronJob controller will attempt to start a Job within this many seconds after the scheduled time. If the window passes, the job is skipped. Useful for clusters with maintenance windows. - successfulJobsHistoryLimit and failedJobsHistoryLimit: Keep these low (e.g., 3 and 1) to avoid accumulating completed Job and Pod objects that consume etcd storage. - restartPolicy: Must be OnFailure or Never for Jobs (not Always). OnFailure allows retries within the same Job; Never creates a new Pod each attempt. To test changes without affecting production, create a copy of the CronJob in a separate namespace or with a different name: ```bash kubectl get cronjob hello-world -o yaml | sed 's/name: hello-world/name: hello-world-test/' | kubectl apply -f - ``` Or, if you are using kustomize, maintain a base and an overlay for test. After applying the test CronJob, manually trigger an immediate run to validate the Job spec without waiting for the schedule: ```bash kubectl create job --from=cronjob/hello-world-test manual-test-001 ``` Then inspect the resulting Job and Pod: ```bash kubectl get jobs -l job=manual-test-001 kubectl get pods -l job-name=manual-test-001 kubectl logs -l job-name=manual-test-001 ``` If the logs show the expected output and the Pod completes, you can promote the change to the production CronJob. Always record the exact change and the verification command in your runbook. A practical sequencing example: if you need to change the schedule from 5 minutes to 10 minutes and update the container image, do it in two separate patches. First, patch the schedule with a single kubectl patch command: ```bash kubectl patch cronjob hello-world -p '{"spec":{"schedule":"*/10 * * * *"}}' ``` Verify the schedule with `kubectl get cronjob hello-world -o jsonpath='{.spec.schedule}'`. Then, patch the image in a separate step after confirming scheduling still works. This limits the blast radius if one change causes unexpected behavior. ## Verification and Diagnostics Once a CronJob is deployed or modified, you need to verify that it is actually executing as intended. The core commands for verification are `kubectl get`, `kubectl describe`, and `kubectl logs`, but you must know which labels and fields to look at because CronJobs create Jobs which create Pods, and each level has its own status. Start with the CronJob status: ```bash kubectl get cronjob hello-world -o wide ``` Look for the LAST SCHEDULE and ACTIVE columns. If LAST SCHEDULE is blank and ACTIVE is 0 for a long time, the controller may be suspended or the schedule may be invalid. Check with: ```bash kubectl get cronjob hello-world -o jsonpath='{.spec.suspend} {.spec.schedule} {.status}' ``` Expected output: `false */5 * * * * {"lastScheduleTime":"2025-03-21T10:00:00Z","active":[]}`. If suspend is true, the CronJob is paused; resume it with `kubectl patch cronjob hello-world -p '{"spec":{"suspend":false}}'`. To see all Jobs created by this CronJob, use the label selector that Kubernetes automatically adds: `job-name` is unique per Job, but you can list by the CronJob owner with: ```bash kubectl get jobs -l job-name ``` However, a better approach is to use the field selector for owner references: ```bash kubectl get jobs -o json | jq '.items[] | select(.metadata.ownerReferences[0].name=="hello-world") | .metadata.name' ``` Or, if you have the Kubernetes CLI `kubectl` with the job name pattern, you can list recent Jobs sorted by creation timestamp: ```bash kubectl get jobs --sort-by=.metadata.creationTimestamp | grep hello-world ``` For a specific Job, inspect its pods and their statuses: ```bash kubectl describe job hello-world-1650000000 ``` This shows events like `Created pod: hello-world-1650000000-abcde` and `Completed`. If the Pod is failing, you will see `Back-off restarting failed container`. To debug a failing pod, get its logs: ```bash kubectl logs ``` If the container has restarted and you want logs from the previous instance, use `--previous`: ```bash kubectl logs --previous ``` A common failure is a non-zero exit code due to a script error. To see the exact exit code: ```bash kubectl get pod -o jsonpath='{.status.containerStatuses[*].state.terminated.exitCode}' ``` For performance or scheduling issues, use `kubectl describe pod ` to check events for messages like `failed to schedule` or `insufficient cpu`. If the CronJob is supposed to create a Job but nothing appears, check the controller's decision by looking at CronJob events: ```bash kubectl describe cronjob hello-world ``` Events may show `Cannot determine if job needs to be started: too many missed start times` if the CronJob was suspended for a long time or the controller was down. In that case, you may need to manually trigger a Job with `kubectl create job --from=cronjob/hello-world manual-$(date +%s)` and then investigate why the controller did not run it automatically. For ongoing monitoring, set up a simple watch on the CronJob status: ```bash kubectl get cronjobs --watch ``` This will show changes as they happen, which is useful during a deployment. ## Failure Modes and Recovery CronJobs fail for a variety of reasons, and each failure leaves specific clues in the Kubernetes object status and events. The most common failure modes and how to recover from them are: 1. **Job not created at scheduled time** - Why: CronJob suspended, controller down, invalid schedule, or missed too many start times. - How to detect: `kubectl get cronjob` shows no recent LAST SCHEDULE, and `kubectl describe cronjob` shows "too many missed start times" or no events. - Recovery: Check suspend state (`kubectl get cronjob -o jsonpath='{.spec.suspend}'`), verify controller pod is running, validate schedule with an online cron parser, and if needed, delete the CronJob and reapply it. Then manually trigger a Job to catch up if appropriate. 2. **Job created but Pod fails immediately with CrashLoopBackOff** - Why: Container command exits non-zero, missing dependencies, or configuration error. - How to detect: `kubectl get pods` shows status CrashLoopBackOff; `kubectl describe pod` shows "Back-off restarting failed container"; `kubectl logs --previous` shows the error. - Recovery: Fix the command or environment, then either update the CronJob template and wait for the next scheduled run, or manually create a Job from the updated CronJob to verify. Then clean up old failed Pods. 3. **Pod runs but exits with success while the actual task fails** - Why: The script inside the container does not propagate errors; it may swallow exit codes or run a command that fails silently. - How to detect: Job completes successfully but expected output is missing or data is corrupt. Check application logs and exit code. - Recovery: Modify the container command to use `set -e` in shell scripts, or ensure the program exits with a non-zero code on failure. Use a health check or post-run verification step within the Job. If possible, run the command manually in a test Pod. 4. **Resource exhaustion or quota exceeded** - Why: CronJob requests more CPU/memory than namespace quota allows, or cluster is overcommitted. - How to detect: `kubectl describe cronjob` shows "failed to create job: pods is forbidden: exceeded quota"; `kubectl get pods` shows Pending. - Recovery: Adjust resource requests and limits, increase namespace quota, or schedule the CronJob during off-peak hours. Also consider using priority classes if available. 5. **Overlapping runs causing data corruption** - Why: concurrencyPolicy set to Allow and Jobs take longer than the interval, resulting in simultaneous runs that may conflict. - How to detect: See multiple active Jobs for the same CronJob (`kubectl get jobs` with ACTIVE > 1), or data anomalies. - Recovery: Set concurrencyPolicy to Forbid or Replace, redesign the workload to be idempotent, or increase the interval. If a corrupted run occurred, restore from backup and manually invoke a clean-up Job. 6. **Deprecated API version breaks apply** - Why: Cluster upgraded and batch/v1beta1 removed. - How to detect: `kubectl apply -f cronjob.yaml` returns error `no matches for kind "CronJob"`. - Recovery: Update manifest to apiVersion: batch/v1 and re-apply. Check other API deprecations with `kubectl api-versions`. For each failure, document the exact command that showed the problem and the command that confirmed recovery. A simple runbook entry might look like: - **Symptom:** CronJob hello-world not running, `kubectl get cronjob` shows SUSPEND=True. - **Action:** Run `kubectl patch cronjob hello-world -p '{"spec":{"suspend":false}}'`. - **Verification:** `kubectl get cronjob hello-world` shows SUSPEND=False and after the next schedule, `kubectl get jobs` shows a new Job. ## Operations Checklist Use this checklist before, during, and after any CronJob change to ensure safety and consistency. Each item includes a concrete command and expected result. The accountable owner for each item should be a named individual, typically the engineer responsible for the service or the on-call operator. Revisit this checklist in a weekly operations review to update any changed procedures. | Step | Command / Action | Expected Result | Owner | Frequency | |------|------------------|-----------------|-------|-----------| | 1. Check cluster version | `kubectl version --short` | Server >= 1.21 for batch/v1 | DevOps Lead | Before any CronJob work | | 2. Verify controller | `kubectl get pods -n kube-system \| grep controller-manager` | Pod Running | Cluster Admin | Weekly | | 3. List CronJobs | `kubectl get cronjobs -n ` | Names, schedules, suspend state | Service Owner | Daily | | 4. Inspect a CronJob | `kubectl describe cronjob -n ` | Events, last schedule, policy | Service Owner | Before change and after | | 5. Snapshot current state | `kubectl get cronjob -o yaml > backup.yaml` | File saved, no secrets | Operator | Before any change | | 6. Make scoped change | `kubectl patch ...` or `kubectl apply -f updated.yaml` | Change applied successfully | Service Owner | When needed | | 7. Verify change | `kubectl get cronjob -o jsonpath='{.spec...}'` | Reflects new value | Operator | Immediately after | | 8. Check Job creation | `kubectl get jobs -n ` after scheduled time | New Job appears | Operator | Next schedule | | 9. Check Pod status | `kubectl get pods -l job-name=` | Completed or Running | Operator | After Job creation | | 10. Check logs | `kubectl logs ` | Expected output, no errors | Service Owner | After completion | | 11. Monitor for recurring failures | `kubectl get jobs --field-selector status.successful=0` | No unexpected failures | Monitoring Team | Daily | | 12. Clean up old Jobs | Set history limits, or manual `kubectl delete job ` | Limited history retained | Service Owner | Weekly | For emergency situations, such as a CronJob triggering a runaway workload, the immediate action is to suspend it: ```bash kubectl patch cronjob -p '{"spec":{"suspend":true}}' ``` This stops future Job creation without deleting the CronJob definition. Then you can safely investigate and delete any active Jobs if necessary: `kubectl delete jobs --all -n ` would remove all Jobs, but be cautious and use label selectors to target only those from the problematic CronJob. ## Common Pitfalls and How to Avoid Them Even experienced Kubernetes users fall into traps with CronJobs. Here are the most common pitfalls, why they happen, and how to avoid or recover from them. **Pitfall 1: Ignoring time zones** Kubernetes schedules use the timezone of the kube-controller-manager, which is typically UTC. If you intend a job to run at 2 AM local time but schedule it as `0 2 * * *`, it will run at 2 AM UTC. From Kubernetes 1.24, you can set `spec.timeZone` to avoid confusion. Always test the schedule with `kubectl get cronjob -o yaml` and verify the last schedule time against your local clock. **Pitfall 2: Not setting concurrencyPolicy** If you omit `concurrencyPolicy`, it defaults to Allow, which can lead to overlapping jobs that corrupt data if the job is not idempotent. For example, a backup job that runs every hour but takes 90 minutes will have overlapping backups, potentially locking the database. Set `concurrencyPolicy: Forbid` unless you absolutely need concurrent runs. **Pitfall 3: Leaving successfulJobsHistoryLimit too high** By default, `successfulJobsHistoryLimit` is 3 and `failedJobsHistoryLimit` is 1, but if you have hundreds of CronJobs, the completed Job and Pod objects can consume massive amounts of etcd storage and clutter `kubectl get jobs`. Set these limits to low numbers or zero if you do not need history. **Pitfall 4: Misunderstanding restartPolicy** The Job's Pod template must set `restartPolicy` to OnFailure or Never. If you set it to Always (the default for Deployments), the Job controller will reject it with an error. Always specify it explicitly. **Pitfall 5: Using a container image that does not exit** If your container runs a long-lived process (like a web server) instead of a short-lived task, the Job will never complete and will eventually hit `activeDeadlineSeconds` if set, otherwise it will hang forever. Ensure your container executes a finite command. **Pitfall 6: Modifying a CronJob in place without testing** Using `kubectl edit` to change a schedule or image directly is risky because you may introduce a typo or forget to record the change. Always keep manifests in version control and apply with `kubectl apply`. For quick experiments, create a copy in a test namespace. **Pitfall 7: Not accounting for missed jobs during maintenance** If you suspend a CronJob for a maintenance window and then resume it, you might miss the jobs that should have run during the suspension if `startingDeadlineSeconds` is not set. Consider whether you need to manually trigger catch-up jobs. ## Conclusion Kubernetes CronJobs are powerful but require disciplined operations. The commands and patterns in this guide give you a solid foundation to inspect, configure, verify, and recover scheduled jobs safely. Remember the core principles: observe before changing, make small reversible changes, protect sensitive data, and always verify the outcome. Start with a small, safe change: list all your CronJobs (`kubectl get cronjobs`), pick one with a non-critical workload, snapshot its current YAML, and test a schedule modification in a separate namespace. Run the verification commands from Section 4, and document the results. As you gain confidence, expand to more complex configurations like time zones and concurrency policies. By following the operational checklist and being aware of common pitfalls, you can keep your scheduled workloads healthy, avoid unexpected failures, and recover quickly when problems arise.