Kubernetes CronJobs are excellent for recurring tasks such as nightly data exports, log rotation, or periodic cleanup. Yet small configuration mistakes can cause overlapping runs, missed schedules, noisy history objects, or jobs that never start. This guide shows practical, safe ways to configure, verify, and troubleshoot CronJobs with concrete YAML and kubectl examples. You will learn how to inventory your environment, choose safe defaults, roll out changes without surprises, validate behavior, recover from failures, and adopt a checklist that keeps operations repeatable. ## Version and Environment Inventory Before changing anything, capture a minimal but complete view of your environment. Version and topology often determine which fields are available and how the controller behaves. ### Commands - Verify client and server versions: kubectl version --short - List CronJob API groups to confirm support: kubectl api-resources | grep -i cronjob - Inventory namespaces and existing CronJobs: kubectl get cronjobs.batch -A - Inspect a specific CronJob deeply: kubectl get cronjob my-cron -n myns -o yaml ### What to capture (constructed examples shown; verify in your environment) Item to capture Example command Example value (constructed) Kubernetes server version kubectl version --short Server Version: v1.28.3 CronJob API kubectl api-resources batch/v1 CronJob Time zone support kubectl explain cronjob.spec.timeZone Field present in v1.27+ Namespace and name n/a myns/my-cron Controller health kubectl get deploy -n kube-system cronjob controller available ### Notes - batch/v1 CronJob is broadly available in modern clusters. If your cluster is very old, confirm whether you are on batch/v1beta1 and plan a migration. - The timeZone field is available in Kubernetes 1.27+. If the field is not present in kubectl explain output, do not use it. ## Safe Configuration Path This section covers common configuration mistakes and safe choices, with practical YAML you can copy and adapt. ## Mistake 1: Allowing overlapping runs by default Symptom: A new run starts while the previous one is still active, causing duplicate work or resource contention. Why it happens: concurrencyPolicy defaults to Allow . Safe choice: Prefer Forbid for idempotent tasks that should not overlap. Use Replace only if the new run must preempt the old one. Bad example (overlaps likely): apiVersion: batch/v1 kind: CronJob metadata: name: overlap-prone spec: schedule: "* * * * *" # every minute jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - name: worker image: busybox:1.36 command: ["/bin/sh","-c","echo starting; sleep 90; echo done"] Fix: spec: concurrencyPolicy: Forbid schedule: "* * * * *" jobTemplate: spec: backoffLimit: 1 template: spec: restartPolicy: OnFailure containers: - name: worker image: busybox:1.36 command: ["/bin/sh","-c","echo starting; sleep 90; echo done"] Expected result: When a minute ticks while the prior run is still sleeping, the new run is skipped. Verify with: kubectl get jobs -l cronjob-name=overlap-prone ## Mistake 2: Invalid restartPolicy in Job template Symptom: Job never schedules Pods; you see validation errors like restartPolicy: Always is not allowed for Jobs . Safe choice: Set restartPolicy to OnFailure (most common) or Never . Fix snippet: spec: jobTemplate: spec: template: spec: restartPolicy: OnFailure # not Always ## Mistake 3: Missed schedules due to too-small startingDeadlineSeconds Symptom: CronJob does not create a Job after a temporary control-plane hiccup, image registry slowness, or clock skew. Why it happens: startingDeadlineSeconds limits the window in which missed runs are started. Too small means the CronJob gives up quickly. For example, if a daily job at 02:00 misses its window because the API server was unreachable for 15 minutes, a startingDeadlineSeconds of 600 (10 minutes) will cause that day's run to be permanently skipped. Safe choice: Set a reasonable window. For hourly jobs, use 3600 seconds (1 hour). For daily jobs, use 86400 seconds (24 hours) or at least 3600. Example fix for an hourly job: spec: schedule: "0 * * * *" startingDeadlineSeconds: 3600 # allow catch-up within an hour ## Mistake 4: Encoding time zone in the cron expression or relying on node time Symptom: Jobs fire at the wrong local time or drift during DST transitions. For instance, a job scheduled at 0 2 * intended for 2 AM EST will run at 2 AM UTC (9 PM EST the previous day) in winter and 3 AM UTC (10 PM EST) in summer if the cluster uses UTC. Safe choice: - If your cluster supports it (Kubernetes 1.27+), use spec.timeZone to set the job time zone explicitly. - If not supported, schedule in UTC and convert business times to UTC in your schedule. Example with timeZone: spec: schedule: "0 2 * * *" timeZone: "America/New_York" # use only if supported in your cluster version ## Mistake 5: Over-retaining or under-retaining job history Symptom: API noise from hundreds of old Job objects or, conversely, not enough history to troubleshoot a failure from last week. Safe choice: Tune limits per CronJob based on schedule frequency. For a daily job, keep 7 successful and 10 failed. For an hourly job, keep 24 successful and 48 failed. Example: spec: successfulJobsHistoryLimit: 7 failedJobsHistoryLimit: 10 jobTemplate: spec: ttlSecondsAfterFinished: 86400 # optional in the Job spec when TTL controller is enabled ## Mistake 6: Unbounded retries on failing Pods Symptom: Pods retry indefinitely and overlap with following schedules, especially when restartPolicy: OnFailure combines with no backoffLimit . Safe choice: Limit retries and runtime. Set backoffLimit to cap Pod restarts, and activeDeadlineSeconds to cap total Job runtime. Example: spec: jobTemplate: spec: backoffLimit: 2 activeDeadlineSeconds: 1200 # 20 minutes hard cap for each job template: spec: restartPolicy: OnFailure With backoffLimit: 2 , a failing Pod restarts at most twice (3 attempts total). After that, the Job is marked failed and the CronJob controller will not retry until the next schedule. ## Mistake 7: Missing service account or permissions Symptom: Pods start but fail with RBAC errors when they need to call the API or access secrets. Safe choice: Assign a minimal ServiceAccount and required Role/RoleBinding in the same namespace. Example snippet: apiVersion: v1 kind: ServiceAccount metadata: name: cron-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: cron-reader rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get","list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: cron-reader-binding subjects: - kind: ServiceAccount name: cron-sa roleRef: kind: Role name: cron-reader apiGroup: rbac.authorization.k8s.io --- apiVersion: batch/v1 kind: CronJob metadata: name: rbac-aware spec: schedule: "*/15 * * * *" concurrencyPolicy: Forbid jobTemplate: spec: template: spec: serviceAccountName: cron-sa restartPolicy: OnFailure containers: - name: worker image: busybox:1.36 command: ["/bin/sh","-c","kubectl get cm || echo no kubectl here"] ## Mistake 8: Using Always image pull for stable images Symptom: Unnecessary image pulls increase latency or fail when registry is down. Safe choice: If you pin to a digest or immutable tag, use IfNotPresent . containers: - name: worker image: myrepo/tool@sha256:abc123... imagePullPolicy: IfNotPresent ## Safer Rollout Habits - Test runs: Use kubectl create job --from=cronjob/ to simulate a scheduled job immediately. - Dry-run validation: Use kubectl apply --dry-run=server -f cron.yaml to catch schema errors. - Staged change: Introduce a new CronJob with -v2 suffix, run it in parallel but suspend the old one ( suspend: true ) for a single window, then finalize the switch. - Suspension control: Temporarily pause a CronJob if needed: kubectl patch cronjob my-cron -p '{"spec":{"suspend":true}}' ## Verification and Diagnostics Use these checks to confirm expected behavior and to quickly isolate problems. ### Core checks - Confirm controller health and scheduling activity: kubectl get cronjob my-cron -n myns kubectl describe cronjob my-cron -n myns - List Jobs created by a CronJob: kubectl get jobs -n myns -l cronjob-name=my-cron - Inspect Pod events and logs for the latest Job: JOB=$(kubectl get jobs -n myns -l cronjob-name=my-cron -o jsonpath='{.items[-1:].metadata.name}') kubectl get pods -n myns -l job-name=$JOB POD=$(kubectl get pods -n myns -l job-name=$JOB -o jsonpath='{.items[0].metadata.name}') kubectl logs -n myns $POD - Validate cron expression parsing and fields: kubectl apply --dry-run=server -f cron.yaml kubectl explain cronjob.spec # confirm available fields ### Common mistakes, symptoms, checks, and fixes Mistake What you see Verify with Recommended fix Overlapping runs (default Allow) Multiple active Jobs kubectl get jobs -l cronjob-name=NAME Set concurrencyPolicy: Forbid or Replace Invalid restartPolicy Job has 0 Pods; validation error kubectl apply --dry-run=server -f Set restartPolicy: OnFailure or Never Too-small startingDeadlineSeconds Missed schedules not backfilled kubectl describe cronjob Increase startingDeadlineSeconds Missing timeZone support Field ignored or apply fails kubectl explain cronjob.spec.timeZone Use UTC schedules or upgrade cluster Excessive history retention Hundreds of Job objects kubectl get jobs -l cronjob-name=NAME Lower successful/failed history limits Unbounded retries Long-running or overlapping failures kubectl describe job ; look at backoff Set backoffLimit and activeDeadlineSeconds RBAC missing Pod logs show Forbidden errors kubectl logs ; kubectl auth can-i Attach minimal ServiceAccount and Role ### Expected results of healthy CronJobs - kubectl get cronjob shows LAST SCHEDULE with a recent timestamp that matches your schedule and time zone strategy. - kubectl get jobs shows either one successful job for each expected period or a currently running one. Statuses progress from 0/1 to 1/1 completions. - kubectl logs of the latest Pod contain expected output without repeated crash loops. ## Manual Trigger and Backfill When you need to test logic without waiting for schedule: - Create a one-off job from the CronJob template: kubectl create job --from=cronjob/my-cron my-cron-adhoctest -n myns - Label and track the ad-hoc job distinctly: kubectl label job my-cron-adhoctest purpose=adhoc -n myns - Delete the ad-hoc job after validation: kubectl delete job my-cron-adhoctest -n myns ## Failure Modes and Recovery Understand what fails, how it looks, and how to recover safely. ### Controller or API outages - Symptom: No new Jobs created at expected times. LAST SCHEDULE stops advancing. - Cause: Control plane disruption or CronJob controller not running. - Recovery: After control plane recovers, ensure startingDeadlineSeconds is large enough to backfill missed runs. Confirm the controller deployment is healthy. ### Clock skew and time zone confusion - Symptom: Schedules drift relative to business time or DST. - Recovery: Prefer UTC schedules or use spec.timeZone if your version supports it. Validate by comparing LAST SCHEDULE to a known UTC clock. ### Image pull errors - Symptom: ImagePullBackOff on Pods. - Recovery: Verify imagePullSecrets , registry availability, and consider IfNotPresent for immutable images. Re-run with an ad-hoc job to confirm. ### RBAC failures - Symptom: Forbidden errors in logs when accessing API resources. - Recovery: Attach a minimal ServiceAccount, Role, and RoleBinding. Test permissions with: kubectl auth can-i get configmaps --as=system:serviceaccount:myns:cron-sa -n myns ### Pod crash loops or timeouts - Symptom: Pods restart repeatedly or exceed execution windows. - Recovery: Set backoffLimit to a reasonable number, add activeDeadlineSeconds , and ensure the job is idempotent. Investigate container logs. ### History bloat - Symptom: Large numbers of completed Jobs cause list operations to slow down. - Recovery: Lower successfulJobsHistoryLimit and failedJobsHistoryLimit ; enable ttlSecondsAfterFinished in the Job template if the TTL controller is available. ### Emergency stop and resume - Temporarily suspend scheduling: kubectl patch cronjob my-cron -n myns -p '{"spec":{"suspend":true}}' - Resume after mitigation: kubectl patch cronjob my-cron -n myns -p '{"spec":{"suspend":false}}' ## Rollback Strategy - Keep a known-good manifest for every CronJob. To roll back, re-apply the previous file explicitly: kubectl apply -f cronjob-prev.yaml - If a field change requires recreation (rare for CronJobs), suspend the current job, create the replacement with a distinct name, validate a test run, then remove the old one. ## Verification After Recovery - Confirm LAST SCHEDULE advances again. - Ensure only one Job runs per period when concurrencyPolicy is Forbid . - Check that history limits retain enough context for post-incident review without bloat. ## Operations Checklist Use this concise checklist during reviews and before/after changes. ### Inventory and prerequisites - [ ] kubectl version --short captured (client and server) - [ ] CronJob API group confirmed ( batch/v1 ) - [ ] timeZone field presence checked with kubectl explain - [ ] Namespace and names recorded ### Configuration safety - [ ] concurrencyPolicy set to Forbid (or Replace with reason) - [ ] restartPolicy set to OnFailure or Never - [ ] startingDeadlineSeconds sized to expected tolerance window - [ ] successfulJobsHistoryLimit and failedJobsHistoryLimit tuned - [ ] backoffLimit and activeDeadlineSeconds reasonable - [ ] serviceAccountName set if API/secret access needed, RBAC minimal - [ ] Image reference immutable, imagePullPolicy appropriate - [ ] suspend set appropriately during rollout or pause windows ### Validation steps - [ ] kubectl apply --dry-run=server passes - [ ] kubectl describe cronjob shows next schedule time expected - [ ] Manual ad-hoc job created and completed for new or risky changes ### Observation and diagnostics - [ ] LAST SCHEDULE updates as expected post-change - [ ] Only expected number of Jobs active per period - [ ] Pod logs inspected for success signal ### Recovery and rollback - [ ] Known-good manifest on hand for immediate rollback - [ ] Ability to suspend/resume verified - [ ] Cleanup of ad-hoc or failed Jobs performed ### Reference signals and actions
SignalWhat it suggestsNext action
LAST SCHEDULE not advancingController or schedule issueCheck controller health; review startingDeadlineSeconds
Many concurrent JobsOverlap allowedSet concurrencyPolicy: Forbid or Replace
ImagePullBackOffRegistry or pull policy issueVerify secrets; consider IfNotPresent for immutable tags
Forbidden in logsRBAC gapAttach ServiceAccount; add minimal Role/Binding
Jobs never finishUnbounded retries or long taskSet backoffLimit and activeDeadlineSeconds
## Conclusion CronJobs are reliable when configured deliberately. The safest path is to inventory versions and fields, adopt non-overlapping execution by default, size retry and deadline limits, and set retention that keeps history useful without causing bloat. Validate changes with server-side dry-run and a one-off job before enabling schedules, and keep a simple rollback plan by retaining known-good manifests. With these practices and the checklist above, you can reduce operational surprises and keep recurring tasks predictable.