## Intro Kubernetes CronJobs run short-lived Pods to perform scheduled work. Unlike long-running Deployments, these Pods often fail silently between runs. Networking misconfigurations are a top cause: DNS resolution, cluster Services and ports, NetworkPolicies, or node- and cloud-level egress controls. Because CronJobs are ephemeral, they can be harder to catch in the act. This guide shows practitioners how to troubleshoot CronJobs networking with minimal blast radius. You will inventory the environment, create a safe diagnostic path, run focused tests for DNS and ports, verify results, and recover cleanly if a change goes wrong. Everything here is intended to be practical and reproducible. ## Version and Environment Inventory Before changing anything, collect facts. This avoids guessing and helps triage at the correct layer. Prerequisites: - kubectl access to the cluster and namespaces in scope - Ability to create a temporary namespace and apply manifests - Basic familiarity with CronJob, Job, Pod concepts Inventory commands: - Identify Kubernetes and kubectl versions: kubectl version --short - List nodes and CNI hints (provider labels can hint at CNI and networking mode): kubectl get nodes -o wide - Confirm CoreDNS is present and record its image version: kubectl -n kube-system get deploy coredns -o yaml | grep -i image - Check if NetworkPolicy is enforced (Calico, Cilium, etc. usually label their components): kubectl get pods -A | grep -Ei "calico|cilium|weave|ovn|antrea" - Note namespaces and existing CronJobs: kubectl get ns kubectl get cronjobs -A For each target CronJob, capture its spec as a backup before making changes: kubectl -n YOUR_NS get cronjob YOUR_CRON -o yaml > YOUR_CRON.backup.yaml Record the following: - Kubernetes version and cloud/on-prem location - CoreDNS version - Whether NetworkPolicy controllers are present - Whether egress is controlled by NetworkPolicy, node firewall, or cloud security groups ## Safe Configuration Path Use a narrow and measurable pilot so you can observe results without impacting production workloads. The steps below create a low-risk diagnostic path. - Create an isolated test namespace: kubectl create ns jobs-test - Create a minimal probing CronJob with explicit labels and conservative policies. This constructed example performs DNS lookups, an external HTTPS request, and a port probe to a Service name. Adjust targets to match your environment. Example manifest: apiVersion: batch/v1 kind: CronJob metadata: name: net-check namespace: jobs-test spec: schedule: "*/10 * * * *" concurrencyPolicy: Forbid startingDeadlineSeconds: 60 successfulJobsHistoryLimit: 1 failedJobsHistoryLimit: 1 jobTemplate: spec: backoffLimit: 0 template: metadata: labels: app: net-check spec: restartPolicy: Never containers: - name: busybox image: busybox:1.36 command: ["sh","-lc t args: - | set -eu echo "DNS test..." nslookup kubernetes.default.svc.cluster.local || exit 12 nslookup example.com || exit 13 echo "HTTP test..." wget -q --spider --timeout=5 https://example.com || exit 14 echo "Port test to a Service..." nc -vz -w5 my-service.my-namespace.svc.cluster.local 5432 || true echo "All checks attempted" Notes: - The image is chosen for simplicity. If a command is missing in your base image, switch tools accordingly or run ad-hoc diagnostics (see below). - The port test uses a hypothetical Service name and port. Replace with a real in-cluster target (for example, postgres.database.svc.cluster.local 5432 ). - backoffLimit: 0 prevents automatic retries that can spam logs. - concurrencyPolicy: Forbid avoids overlapping runs when you test repeatedly. - Optionally add a targeted NetworkPolicy to prove or disprove DNS/egress hypotheses. Start narrow, then widen. This example allows DNS to CoreDNS (by namespace) and outbound TCP/443 to the internet. Tighten ipBlock to your real egress CIDRs where possible. apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-dns-and-https namespace: jobs-test spec: podSelector: matchLabels: app: net-check policyTypes: - Egress egress: - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 - protocol: TCP port: 53 - to: - ipBlock: cidr: 0.0.0.0/0 ports: - protocol: TCP port: 443 - Apply and run an immediate one-off Job based on the CronJob template so you do not wait for the schedule: kubectl -n jobs-test apply -f net-check.yaml kubectl -n jobs-test create job --from=cronjob/net-check net-check-now - Observe output and exit code: kubectl -n jobs-test logs job/net-check-now kubectl -n jobs-test get job net-check-now -o jsonpath='{.status.succeeded}{"\n"}' - For interactive probing, run an ad-hoc diagnostic Pod and test in-cluster name resolution and connectivity: kubectl -n jobs-test run diag --rm -it --image=busybox:1.36 --restart=Never -- sh Inside the shell: cat /etc/resolv.conf nslookup kubernetes.default.svc.cluster.local nslookup example.com wget -q --spider --timeout=5 https://example.com nc -vz -w5 my-service.my-namespace.svc.cluster.local 5432 ## Verification and Diagnostics Interpret what you see using a focused sequence of checks. Change only one variable at a time. ### 1. Confirm the Job actually ran Expected when successful: status.succeeded=1 , backoff not exceeded, and Pod phase Succeeded. kubectl -n jobs-test get jobs kubectl -n jobs-test describe job net-check-now ### 2. Check Pod events and logs Look for timeouts, connection refused, or name resolution errors. kubectl -n jobs-test get pods -l job-name=net-check-now kubectl -n jobs-test describe pod POD_NAME kubectl -n jobs-test logs POD_NAME ### 3. DNS checks Expected: few or no SERVFAIL or timeout lines correlated with your Job start time. - Inspect resolver configuration from inside a similar Pod: kubectl -n jobs-test exec -it DEPLOYMENT_OR_POD -- cat /etc/resolv.conf - Typical ndots=5 means a short name like example.com triggers multiple search path lookups first. Use a trailing dot for absolute names ( example.com. ) or use full FQDNs to avoid search path delays in short-lived Pods. - Check CoreDNS health and query errors: kubectl -n kube-system logs deploy/coredns --tail=200 | grep -Ei "timeout|servfail|refused|dns" || true ### 4. Service and Endpoint checks (in-cluster targets) Expected: The Service has at least one endpoint. If endpoints are empty, the backing Pods are not ready or selectors are wrong. kubectl -n my-namespace get svc my-service -o wide kubectl -n my-namespace get endpoints my-service -o wide ### 5. NetworkPolicy checks Expected: If there is a default-deny egress policy, you must explicitly allow DNS and the required destinations. - List policies in both the CronJob namespace and target namespace: kubectl -n jobs-test get networkpolicy kubectl -n my-namespace get networkpolicy - Examine rules that might select the CronJob Pods by label or namespace: kubectl -n jobs-test describe networkpolicy NAME ### 6. Egress and node or cloud firewalls Expected: Nodes can reach required external addresses on necessary ports. - If DNS works but HTTPS to the internet fails, check whether your environment restricts egress: - Cloud: verify the node security group or firewall rules allow outbound TCP/443 and UDP/TCP/53. - On-prem: confirm node firewalls permit UDP/TCP/53 to CoreDNS and required external ports, and that NAT is available. ### 7. kube-proxy and routing sanity Expected: No repeated sync or iptables/ipvs errors at Job run time. - If a ClusterIP Service is unreachable but endpoints exist, the issue can be kube-proxy or node routing. Check kube-proxy logs for errors (namespace usually kube-system): kubectl -n kube-system logs -l k8s-app=kube-proxy --tail=200 ### Common symptoms, likely causes, and what to check Symptom or log string Likely cause What to check no such host DNS failure or blocked DNS egress nslookup inside Pod; CoreDNS logs; NetworkPolicy for UDP/TCP 53 i/o timeout Egress blocked or remote endpoint unreachable Cloud/node firewall; ipBlock rules; traceroute alternative (curl/wget -m) connection refused Target port closed or wrong Service/port Service .spec.ports, Endpoints, target app listener Empty Endpoints Selectors mismatch or Pods not Ready Service selector; Pod labels; readiness gates BackoffLimitExceeded Job could not succeed within retries Underlying network failure; too short timeout in script ### Example expected outputs (constructed) - DNS success: nslookup example.com Server: 10.96.0.10 Address: 10.96.0.10:53 Name: example.com Address: 93.184.216.34 - HTTPS success: wget -q --spider --timeout=5 https://example.com returns exit code 0 - Service port probe (hypothetical): nc -vz -w5 postgres.database.svc.cluster.local 5432 prints succeeded or times out if blocked ### Tightening or relaxing scope safely - If DNS fails, first allow only UDP/TCP 53 to kube-system (CoreDNS). Do not open 0.0.0.0/0 until you prove it is needed. - If external HTTPS is required, allow only known CIDRs or hostnames through a proxy if available. - Always label your CronJob Pod template (for example app: net-check ) so NetworkPolicies can target precisely. ## Failure Modes and Recovery Plan for safe exits during testing and rollout. ### 1. Pause or stop runs quickly - Suspend a CronJob to prevent new Jobs while you investigate: kubectl -n jobs-test patch cronjob net-check -p '{"spec": {"suspend": true}}' - Delete pending Jobs created by schedule: kubectl -n jobs-test delete job -l job-name=net-check ### 2. Roll back manifests - Restore a CronJob from the backup you created earlier: kubectl -n YOUR_NS apply -f YOUR_CRON.backup.yaml - Revert NetworkPolicy by deleting the test policy or re-applying the prior version: kubectl -n jobs-test delete networkpolicy allow-dns-and-https ### 3. Contain blast radius - Use backoffLimit: 0 and short timeouts (for example wget --timeout=5 ) in test Jobs to avoid long hangs. - Prefer concurrencyPolicy: Forbid to block overlaps that might stress downstream services. ### 4. Verify recovery - After a rollback, run a one-off Job again to confirm the system behaves as before: kubectl -n jobs-test create job --from=cronjob/net-check net-check-verify kubectl -n jobs-test logs job/net-check-verify - Ensure your critical production CronJobs are unaffected: kubectl -n PROD_NS get jobs --sort-by=.metadata.creationTimestamp | tail -n 5 ### Typical root causes and remedies Root cause Minimal remedy Verification step Default-deny egress policy blocks DNS Allow UDP/TCP 53 egress to kube-system nslookup kubernetes.default.svc.cluster.local succeeds Wrong Service name or port Correct Service DNS or port and redeploy nc -vz to corrected name:port succeeds External egress blocked Open TCP/443 (or required port) to allowed CIDRs wget --spider https://target.example returns exit 0 Resolver search path delays Use FQDNs or trailing dot; reduce lookups DNS query latency drops; Job exits earlier Missing endpoints due to labels Fix Service selector to match Pods kubectl get endpoints shows ready addresses ## Operations Checklist Use this as a repeatable runbook. Replace placeholders with your values.
StepCommand or actionExpected result
Backup CronJobkubectl -n NS get cronjob NAME -o yaml > NAME.bak.yamlBackup file created
Create test nskubectl create ns jobs-testNamespace exists
Apply test CronJobkubectl -n jobs-test apply -f net-check.yamlCronJob created
One-off runkubectl -n jobs-test create job --from=cronjob/net-check net-check-nowJob created
Get logskubectl -n jobs-test logs job/net-check-nowDNS/HTTP checks visible
Inspect Pod eventskubectl -n jobs-test describe pod PODErrors or successes clear
Check CoreDNS logskubectl -n kube-system logs deploy/coredns --tail=200No DNS timeouts or SERVFAIL at run time
Verify Service endpointskubectl -n APP_NS get endpoints SVCEndpoints not empty
Apply egress policykubectl -n jobs-test apply -f allow-dns-and-https.yamlPolicy in place
Rerun and verifykubectl -n jobs-test create job --from=cronjob/net-check net-check-againSucceeds or clearer failure
Suspend if neededkubectl -n jobs-test patch cronjob net-check -p '{"spec":{"suspend":true}}'New Jobs stop scheduling
Roll back policykubectl -n jobs-test delete networkpolicy allow-dns-and-httpsPolicy removed
## Conclusion CronJobs are uniquely sensitive to networking friction because they are short-lived and scheduled. The fastest path to root cause is to start with a narrow, observable pilot: inventory your environment, run a safe diagnostic CronJob, and test DNS, Service endpoints, and egress in that order. Add or adjust NetworkPolicies with precision, prefer absolute domain names to avoid resolver pitfalls, and keep backoff and timeouts conservative during tests. If a change backfires, suspend the CronJob, revert the policy or spec from backup, and verify with a one-off run. Adopt the checklist in this guide as your runbook. Each time you add a new CronJob with network dependencies, create a short diagnostic variant, prove connectivity in isolation, and only then schedule it. This keeps incidents small, fixes fast, and outcomes predictable.