Kubernetes ReplicaSet Troubleshooting with Practical Examples

Intro

ReplicaSets are a core Kubernetes controller that ensure a specified number of identical pod replicas are running at any given time. When pods fail to start, scale unexpectedly, or enter crash loops, a ReplicaSet is often involved. Troubleshooting these issues requires a systematic approach: observe current state, inspect events and logs, identify the failing component, apply a minimal fix, and verify recovery.

This article is written for developers, DevOps consultants, and technical startup teams who need to diagnose and resolve ReplicaSet-related problems in real clusters. It focuses on the practical commands, expected outputs, failure signals, and recovery decisions you need when a ReplicaSet misbehaves. By the end, you will know how to inspect ReplicaSets and their pods, interpret key conditions, fix common issue like image pull failures, crash loops, and selector mismatches, and prevent recurrence.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery steps. Every troubleshooting command in this guide is read-only unless explicitly marked as a change.

Version and Environment Inventory

Before touching anything, confirm what you are working with. A ReplicaSet's behavior depends on the Kubernetes version, the API group, and the namespace context. Check these first.

Check Kubernetes version and API availability

kubectl version --short

Example output:

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

ReplicaSets have been stable since Kubernetes 1.9 (apps/v1). If you are on an older cluster (before 1.9), you may need to use extensions/v1beta1 or apps/v1beta1. Starting from Kubernetes 1.16, the old API versions are removed. Verify with:

kubectl api-resources | grep replicaset

Expected:

replicasets                       rs           apps/v1                                true         ReplicaSet

Note the API group (apps/v1) and whether the resource is namespaced (true). All ReplicaSet operations are scoped to a namespace.

Inventory existing ReplicaSets

List all ReplicaSets in the current namespace:

kubectl get rs

Example output:

NAME                 DESIRED   CURRENT   READY   AGE
frontend-rs         3         3         3       10m
backend-rs          2         2         1       2h

The columns:

• DESIRED: number of replicas the ReplicaSet wants.
• CURRENT: number of replicas that have been created.
• READY: number of replicas that are ready to serve traffic.
• AGE: how long the ReplicaSet has existed.

A common first signal is a mismatch between DESIRED and READY. For example, backend-rs above wants 2 but only 1 is ready. We will dig into why in a moment.

To see ReplicaSets across all namespaces:

kubectl get rs -A

Check details of a specific ReplicaSet:

kubectl describe rs frontend-rs

Look for Events at the bottom, which show recent controller actions like pod creation or deletion. Also note the Selector and Pod Template; these must match the pods it manages.

Prerequisites for troubleshooting

You need:

• kubectl configured with access to the cluster.
• jq or yq for parsing JSON/YAML output (optional but helpful).
• Access to container logs, either directly or via a logging platform.
• Permissions to read ReplicaSet, Pod, and Event resources in the target namespace. Read-only is enough for diagnosis.
• No secrets in your command lines; use environment variables or temporary files for sensitive data.

A safe initial inventory command is:

kubectl get rs,po -o wide

This shows ReplicaSets and their pods with node placement, IPs, and status. Observe before you change anything.

Safe Configuration Path

When a ReplicaSet is misbehaving, the natural impulse is to delete and recreate it. That can cause an outage, especially if the ReplicaSet is managed by a Deployment. Instead, follow a safe path: understand the current configuration, identify the exact mismatch, and make the smallest possible change with a known rollback.

Read the current manifest

Export the live manifest of a ReplicaSet:

kubectl get rs frontend-rs -o yaml > frontend-rs-live.yaml

Review it carefully. Look for:

• spec.replicas: the desired count.
• spec.selector: the label selector used to find pods to manage.
• spec.template.metadata.labels: labels applied to new pods. These must match the selector.
• spec.template.spec.containers[].image: container images and tags.
• spec.template.spec.containers[].resources: CPU/memory requests and limits.
• spec.template.spec.containers[].env: environment variables (check for typos or missing values).

A common mistake is a selector mismatch. For example, if the selector is app: frontend but the pod template labels are app: web, pods are created but never adopted, leading to orphaned pods and a ReplicaSet that reports 0 ready. Here is a broken snippet:

spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: web

To fix this, update the template label to app: frontend or update the selector. Because selectors are immutable after creation, you may need to delete and recreate the ReplicaSet, but only after confirming it is not managed by a Deployment (see next section).

Check if the ReplicaSet is owned by a Deployment

Deployments create ReplicaSets automatically and manage their lifecycle. If you directly edit or delete a ReplicaSet owned by a Deployment, the Deployment controller will recreate it or adjust it back. To find the owner:

kubectl get rs frontend-rs -o jsonpath='{.metadata.ownerReferences}' | jq

If there is an owner reference to a Deployment (kind: Deployment), make changes at the Deployment level instead:

kubectl edit deployment frontend-deployment

This method ensures the Deployment's rollout history is preserved and you can rollback with kubectl rollout undo deployment frontend-deployment.

Make a small, testable change

When adjusting configuration, change only one field at a time. For instance, if you need to update the image tag, do that alone, then check pod status. Here is an example using a patch to update the image for a standalone ReplicaSet:

kubectl patch rs frontend-rs -p '{"spec":{"template":{"spec":{"containers":[{"name":"web","image":"nginx:1.25.3"}]}}}}'

If the ReplicaSet is managed by a Deployment, patch the Deployment instead:

kubectl set image deployment/frontend-deployment web=nginx:1.25.3

Then watch the rollout:

kubectl rollout status deployment/frontend-deployment

Expected output on success:

deployment "frontend-deployment" successfully rolled out

If the rollout hangs, investigate the new ReplicaSet and pods as described in Failure Modes.

Keep local tests small

Before applying a change to a production cluster, test it in a local or development namespace. For example, create a minimal ReplicaSet manifest and apply it to a test namespace:

cat <<EOF | kubectl apply -n test -f -
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: test-rs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: nginx
        image: nginx:1.25.3
        ports:
        - containerPort: 80
EOF

Then verify with kubectl get rs,pods -n test. If it works, you can confidently apply the equivalent change to production. For accessing the pod locally, use kubectl port-forward to verify it serves traffic:

kubectl port-forward rs/test-rs 8080:80

Then curl localhost:8080 should return the nginx welcome page.

Verification and Diagnostics

After making a change, or when you first notice a problem, you need to verify the actual state of the ReplicaSet and its pods. This section provides a systematic diagnostic flow with concrete commands and expected outputs.

Step 1: Check ReplicaSet status details

Use kubectl describe to see conditions and events:

kubectl describe rs frontend-rs

Look for a Conditions section (present in newer Kubernetes versions) and the Events list. Example snippet:

Conditions:
  Type             Status  Reason
  ----             ------  ------
  ReplicaFailure   True    FailedCreate

FailedCreate typically means the ReplicaSet controller could not create a pod. Events will show the error, such as quota exceeded or invalid image name.

If conditions are not shown, check events directly:

kubectl get events --field-selector involvedObject.name=frontend-rs --sort-by=.lastTimestamp

Step 2: Inspect pods owned by the ReplicaSet

List pods with their owner references:

kubectl get pods -l app=frontend -o wide

Note the STATUS column: Pending, Running, CrashLoopBackOff, ImagePullBackOff, ErrImagePull, etc. If pods are missing, check if the ReplicaSet created any:

kubectl get pods --selector=app=frontend

If no pods appear but the ReplicaSet's DESIRED is > 0, examine the ReplicaSet events for reasons (e.g., FailedCreate due to quota or permission).

For a pod that is not ready, inspect details:

kubectl describe pod <pod-name>

Focus on:

• Conditions: PodScheduled, Initialized, ContainersReady, Ready.
• Events: messages about image pulling, container start failures, probe failures, etc.

Step 3: View container logs

For a running pod, get logs:

kubectl logs <pod-name> -c <container-name>

If the container has crashed and restarted, use --previous to see logs from the last terminated instance:

kubectl logs <pod-name> -c <container-name> --previous

Example output indicating a crash due to missing configuration:

panic: unable to open config file: /etc/app/config.yaml

For pods in CrashLoopBackOff, logs often reveal the root cause. If the container exits immediately without logs, check the container's exit code:

kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'

Common exit codes: 1 (application error), 2 (misuse of shell builtins), 137 (SIGKILL, often OOM), 143 (SIGTERM, graceful shutdown).

Step 4: Verify resource usage and node conditions

Pods may fail to start due to insufficient resources. Check node capacity and requests:

kubectl top nodes

And pod resource usage:

kubectl top pods

If you suspect a scheduling issue, describe the pod and look for events like:

Warning  FailedScheduling  0/3 nodes are available: 3 Insufficient cpu.

Then reduce replicas or resource requests, or scale nodes.

Step 5: Validate selectors and labels

A silent failure mode is a ReplicaSet that reports READY=0 because pods aren't matching the selector. Verify labels on pods:

kubectl get pods -l app=frontend --show-labels

Compare with the ReplicaSet's selector:

kubectl get rs frontend-rs -o jsonpath='{.spec.selector}'

If labels don't match, the pods might be orphaned. You can adopt them by updating their labels (if appropriate) or fix the ReplicaSet's template and trigger a rollout.

Failure Modes and Recovery

This section covers the most common ReplicaSet failure modes with symptoms, diagnosis, and recovery steps. Always verify the fix and document what you did.

1. ImagePullBackOff or ErrImagePull

Symptom: Pods stay in Pending or ContainerCreating, then show ImagePullBackOff or ErrImagePull.

Diagnosis:

kubectl describe pod <pod-name>

Look for events like:

Warning  Failed     2m (x4 over 5m)  kubelet            Failed to pull image "nginx:1.25.3": rpc error: code = NotFound desc = failed to pull and unpack image ... manifest for nginx:1.25.3 not found: manifest unknown

This means the image tag doesn't exist or the registry is unreachable. Also check authentication errors if using a private registry: no basic auth credentials or unauthorized: authentication required.

Recovery

• Verify the image name and tag. Use a known-good tag, e.g., nginx:1.25.3-alpine.
• If using a private registry, ensure the pod has an imagePullSecret:

spec:
  template:
    spec:
      imagePullSecrets:
      - name: regcred

Create the secret if needed:

kubectl create secret docker-registry regcred \
  --docker-server=myregistry.example.com \
  --docker-username=myuser \
  --docker-password=mypassword \
  [email protected]

Then update the ReplicaSet or Deployment. For a Deployment, the rollout will create a new ReplicaSet with the fixed template.

Verification: Watch pods until they become Running and ready:

kubectl get pods -w

Then check the ReplicaSet status:

kubectl get rs frontend-rs

Expect DESIRED == CURRENT == READY.

2. CrashLoopBackOff

Symptom: Pods start but immediately exit, and restart count increases. Status shows CrashLoopBackOff.

Diagnosis:

kubectl logs <pod-name> --previous

Examine the application error. Also check exit code:

kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState.terminated.exitCode}'

Common causes:

• Missing configuration file or environment variable.
• Application bug causing panic.
• Liveness probe failing because app is not ready in time.
• Resource limit too low causing OOM kill.

Recovery:

• Fix the underlying issue (update config map, secret, image, or resource limits).
• For probe issues, adjust initial delay or failure threshold. For example, increase initialDelaySeconds from 0 to 10 for a slow-starting app.
• If the app is stateless, delete the pod to force a restart: kubectl delete pod <pod-name> (the ReplicaSet will recreate it). But this is a temporary fix; address the root cause.

Verification: After fixing, watch the restart count stabilize:

kubectl get pods -w

If restarts stop, the issue is resolved.

3. ReplicaSet not creating pods (FailedCreate)

Symptom: ReplicaSet's DESIRED > CURRENT, and events show FailedCreate.

Diagnosis:

kubectl describe rs <rs-name>

Look for events indicating reasons:

• exceeded quota: resource quota limits in namespace.
• forbidden: RBAC permissions missing.
• cannot set blockOwnerDeletion: owner reference issue.
• invalid pod spec: misconfiguration.

Recovery:

• If quota exceeded, either increase namespace quota or reduce resource requests/replicas.
• If RBAC forbidden, grant the necessary permissions to the controller's service account.
• If invalid pod spec, fix the template (e.g., wrong field name, missing required field).

Verification: Monitor events and pod count:

kubectl get events --sort-by=.lastTimestamp | tail -20
kubectl get rs <rs-name>

4. Pods are created but never become ready

Symptom: READY column shows a fraction or 0, e.g., 0/1.

Diagnosis:

kubectl describe pod <pod-name>

Check Conditions for Ready and ContainersReady. Events might show readiness probe failures. Also check logs for application startup errors.

Recovery:

• If readiness probe is misconfigured (wrong path, port, or initial delay), fix it.
• If the app takes longer to start, increase initialDelaySeconds or failureThreshold.
• If a dependency is missing (database, service), ensure it is available.

Verification: After applying the fix, monitor:

kubectl rollout status deployment/<deployment-name>   # if managed by Deployment

or for standalone ReplicaSet:

kubectl get rs <rs-name> -w

5. ReplicaSet selector mismatch leading to orphaned pods

Symptom: ReplicaSet thinks it has no pods (READY=0), but you see pods with similar labels running. Also kubectl get rs might show CURRENT=0 while DESIRED=3.

Diagnosis:

kubectl get rs <rs-name> -o jsonpath='{.spec.selector}'
kubectl get pods --show-labels

Compare labels. If pods have different labels, they are not selected.

Recovery:

• If the ReplicaSet is standalone, you can either update the pod labels to match the selector (if you want to adopt them) or delete the ReplicaSet and recreate with correct selector/template.
• If the ReplicaSet is owned by a Deployment, update the Deployment's pod template labels. The Deployment will create a new ReplicaSet with matching selector and gradually scale down the old one.
• Be careful: changing selectors can cause temporary duplicate pods. Use kubectl get pods -w to monitor.

Verification: Ensure the ReplicaSet now shows READY == DESIRED.

General recovery principles

• Always take a snapshot before changes: kubectl get rs <name> -o yaml > rs-backup.yaml.
• For Deployments, use rollback: kubectl rollout undo deployment/<name> to revert to a previous working revision.
• Document each change and its verification in an incident log.
• If a ReplicaSet is stuck and not managed by a Deployment, you can delete it, but only after ensuring its pods are safe or will be recreated by another controller.

Operations Checklist

Use this checklist when troubleshooting any ReplicaSet issue. It consolidates the key actions into a repeatable workflow.

1. Observe without changes

• [ ] Run kubectl get rs -n <namespace> to see ReplicaSets and their desired/current/ready counts.
• [ ] Run kubectl get pods -n <namespace> -o wide to see pod status and node distribution.
• [ ] Run kubectl describe rs <rs-name> to view conditions and events.
• [ ] For any failing pod, run kubectl describe pod <pod-name> and kubectl logs <pod-name> --previous if needed.

2. Identify the failure mode

• [ ] Check for ImagePullBackOff: image name/tag errors, registry auth.
• [ ] Check for CrashLoopBackOff: application errors, exit codes, probe misconfigurations.
• [ ] Check for FailedCreate: quota, RBAC, invalid spec.
• [ ] Check for readiness failures: probe settings, dependency availability.
• [ ] Check for selector mismatches: labels on pods vs selector.

3. Plan a minimal fix

• [ ] Back up current manifest: kubectl get rs <rs-name> -o yaml > rs-backup.yaml.
• [ ] If managed by Deployment, modify the Deployment, not the ReplicaSet.
• [ ] Change one field at a time (image, env, resources, probe, labels).
• [ ] Use kubectl apply or kubectl patch with a scoped change.

4. Apply and verify

• [ ] For Deployments, run kubectl apply -f deployment.yaml and kubectl rollout status deployment/<name>.
• [ ] For standalone ReplicaSets, run kubectl apply -f rs.yaml and then kubectl get pods -w to watch pods.
• [ ] Check logs of new pods: kubectl logs <pod-name>.
• [ ] Confirm ReplicaSet READY count matches DESIRED.

5. Document and monitor

• [ ] Record the original error, the change made, and the verification result.
• [ ] Set up alerts on ReplicaSet status if possible (e.g., kubectl get rs via monitoring tool).
• [ ] Review resource quotas and cluster capacity if the issue was resource-related.
• [ ] Consider adding readiness/liveness probes with appropriate delays to prevent future crash loops.

Example checklist entry for a real incident

Context: backend-rs had READY = 1 but DESIRED = 2. One pod was in CrashLoopBackOff.

• Observed with kubectl get rs backend-rs and kubectl describe pod backend-rs-abcde.
• Found logs: Error: unable to connect to database at 10.96.0.10:5432.
• Diagnosis: database service IP changed; environment variable in pod was stale.
• Fix: updated ConfigMap backend-config with correct DB host, then restarted pods via kubectl rollout restart deployment backend-deployment.
• Verified: kubectl rollout status deployment backend-deployment succeeded; kubectl get rs backend-rs showed READY = 2.

Conclusion

ReplicaSet troubleshooting is a fundamental skill for anyone operating Kubernetes. By following a structured approach, you can quickly move from symptom to resolution while minimizing risk. Remember these principles:

• Observe first: use read-only commands (get, describe, logs) to gather facts.
• Isolate the fault: determine whether it is image pulling, container crashing, scheduling, or selector-related.
• Apply minimal changes: patch one thing at a time, and always back up the current state.
• Verify recovery: ensure the ReplicaSet's ready count matches desired and pods are stable.
• Document and improve: record what happened and adjust probes, resource requests, or rollout strategies to prevent recurrence.

As a next step, pick one ReplicaSet in your cluster and run the diagnostic checklist. Practice on a non-production namespace first. For example, create a ReplicaSet with a deliberately wrong image tag, then work through the failure mode and recovery steps. This hands-on experience will build confidence for real incidents.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.