GitLab CI/CD Kubernetes Deployment Troubleshooting: Practical Examples

Intro

Deploying to Kubernetes from GitLab CI/CD becomes reliable when you understand the four moving parts: the runner, the kubeconfig, registry authentication, and rollout verification. This guide gives you concrete fixes for the most frequent failures and copy‑paste snippets you can adapt today. You’ll learn how to create a least‑privilege kubeconfig, diagnose pipeline and cluster errors, verify rollouts, and run a small pilot before scaling.

Workflow Overview

A clear, step‑by‑step path from commit to healthy pods keeps problems isolated to a single stage:

1. Build – produce a container image tagged with the commit SHA.
2. Push – authenticate to the registry and push the image.
3. Configure – supply kubectl and a kubeconfig that grants only the needed permissions.
4. Apply – run kubectl apply or helm upgrade --install.
5. Wait – watch the rollout status with a timeout.
6. Verify – run a quick smoke test against the Service or Ingress.
7. Roll back – revert to the last good ReplicaSet if verification fails.

Keep each step in its own job so you can see exactly where a failure occurs.

Setup: runners and kubeconfig

Service account and RBAC (namespace‑scoped)

Create a dedicated service account with the minimum rights required in the target namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: demo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-deployer
  namespace: demo
rules:
- apiGroups: ["", "apps", "extensions
esources: ["pods", "deployments", "replicasets", "services", "configmaps", "secrets
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete
d---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer
  namespace: demo
subjects:
- kind: ServiceAccount
  name: ci-deployer
  namespace: demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ci-deployer

Generate a short‑lived token (Kubernetes 1.24+):

kubectl -n demo create token ci-deployer > ci-deployer.token

Build a kubeconfig that points to your API server and includes the CA data, then store the full YAML as a protected, masked CI variable named KUBE_CONFIG_B64 (base64‑encoded) or KUBE_CONFIG_YAML (multiline).

Writing kubeconfig in the job

# .gitlab-ci.yml (kubeconfig snippet)
before_script:
  - mkdir -p ~/.kube
  - |
    if [ -n "$KUBE_CONFIG_B64" ]; then
      echo "$KUBE_CONFIG_B64" | base64 -d > ~/.kube/config
    else
      echo "$KUBE_CONFIG_YAML" > ~/.kube/config
    fi
  - chmod 600 ~/.kube/config

Runner image choice

Use a job image that already contains kubectl (for example bitnami/kubectl:latest) or install kubectl at runtime. If you build images with Docker‑in‑Docker, the runner must run in privileged mode. When privileged runners are not an option, switch to a rootless builder such as Kaniko or BuildKit.

Reference pipeline with build, push, and deploy

stages: [build, deploy]

variables:
  IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
  KUBE_NAMESPACE: "demo"

# Build and push image using Docker-in-Docker (requires privileged runner)
build:
  stage: build
  image: docker:25
  services:
    - name: docker:25-dind
      command: ["--mtu=1460\
  variables:
    DOCKER_TLS_CERTDIR: /certs
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - docker build -t "$IMAGE" .
    - docker push "$IMAGE"
  rules:
    - if: "$CI_COMMIT_BRANCH"

# Deploy using kubectl apply and wait for rollout
deploy:
  stage: deploy
  image: bitnami/kubectl:latest
  before_script:
    - mkdir -p ~/.kube
    - if [ -n "$KUBE_CONFIG_B64" ]; then echo "$KUBE_CONFIG_B64" | base64 -d > ~/.kube/config; else echo "$KUBE_CONFIG_YAML" > ~/.kube/config; fi
    - chmod 600 ~/.kube/config
  script:
    # Ensure namespace exists
    - kubectl get ns "$KUBE_NAMESPACE" || kubectl create ns "$KUBE_NAMESPACE"

    # Create or update image pull secret for the namespace (once per namespace)
    - |
      kubectl -n "$KUBE_NAMESPACE" create secret docker-registry gitlab-regcred \
        --docker-server="$CI_REGISTRY" \
        --docker-username="$CI_REGISTRY_USER" \
        --docker-password="$CI_REGISTRY_PASSWORD" \
        --docker-email="[email protected]" \
        --dry-run=client -o yaml | kubectl apply -f -

    # Apply manifests (deployment uses imagePullSecrets)
    - kubectl -n "$KUBE_NAMESPACE" apply -f k8s/

    # Set the container image to the commit SHA tag (optional patch)
    - kubectl -n "$KUBE_NAMESPACE" set image deploy/myapp myapp-container="$IMAGE" --record=true || true

    # Wait for rollout with timeout
    - kubectl -n "$KUBE_NAMESPACE" rollout status deploy/myapp --timeout=120s

    # Simple smoke check against ClusterIP via busybox curl (optional)
    - kubectl -n "$KUBE_NAMESPACE" run tmp-curl --image=busybox:1.36 --restart=Never --rm -it -- curl -sS myapp:8080/healthz
  environment:
    name: demo/$CI_COMMIT_REF_NAME
    url: https://demo.example.com
  rules:
    - if: "$CI_COMMIT_BRANCH == \"main\""

Example Deployment manifest (k8s/deploy.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      serviceAccountName: ci-deployer
      imagePullSecrets:
        - name: gitlab-regcred
      containers:
        - name: myapp-container
          image: registry.example.com/group/project:CHANGE_ME
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
  ports:
    - port: 8080
      targetPort: 8080

Troubleshooting: pipeline failures

Troubleshooting: image pull errors

Pod shows ErrImagePull or ImagePullBackOff:

1. Wrong image tag

• Check the exact tag in the Deployment:

     kubectl -n demo get deploy myapp -o jsonpath='{.spec.template.spec.containers[0].image}'

• Prefer immutable tags such as $CI_COMMIT_SHORT_SHA.

1. Registry authentication missing

• Ensure a docker-registry secret exists in the same namespace and is referenced by imagePullSecrets or the service account.
• Recreate the secret if credentials changed:

     kubectl -n demo delete secret gitlab-regcred --ignore-not-found
     kubectl -n demo create secret docker-registry gitlab-regcred \
       --docker-server="$CI_REGISTRY" \
       --docker-username="$CI_REGISTRY_USER" \
       --docker-password="$CI_REGISTRY_PASSWORD" \
       --docker-email="[email protected]"

1. Policy and caching surprises

• If you use :latest, set imagePullPolicy: Always to avoid stale images.
• Better: pin to the commit SHA and keep imagePullPolicy: IfNotPresent.
• Secrets are namespace‑scoped; create the same pull secret in each target namespace.

To confirm the fix, delete the failing pod and let the ReplicaSet recreate it, or trigger a rollout restart:

kubectl -n demo rollout restart deploy/myapp

Troubleshooting: rollout checks and debugging

Automate the wait and enrich failure logs:

kubectl -n demo rollout status deploy/myapp --timeout=120s || {
  echo "Rollout did not complete in time"
  kubectl -n demo get pods -o wide
  kubectl -n demo describe deploy/myapp
  kubectl -n demo get events --sort-by=.lastTimestamp | tail -n 50
  exit 1
}

If rollout fails or pods crash:

• Describe and events – kubectl -n demo describe pod <name> shows image‑pull, scheduling, and probe issues. kubectl -n demo get events --sort-by=.lastTimestamp surfaces recent errors.
• Logs – kubectl -n demo logs deploy/myapp --all-containers=true --tail=200 for application errors.
• Probes and resources – Readiness failures block rollout. Verify the probe path and port; increase initialDelaySeconds if the app starts slowly. Check CPU/memory requests and limits; OOMKilled means the memory limit is too low.
• Services and Ingress – Ensure Service selectors match pod labels exactly. Confirm targetPort matches the container port. For Ingress, validate host, TLS, and that the Service backends are healthy.
• ConfigMaps and Secrets – Missing keys or wrong mounts often cause CrashLoopBackOff. Compare the running pod spec to the manifest: kubectl -n demo get pod <name> -o yaml.

Fast rollback:

kubectl -n demo rollout undo deploy/myapp

Scripted safety checks in CI

Add preflight and post‑deploy checks to reduce guesswork:

# Validate manifests before apply
kubectl -n demo apply --dry-run=client -f k8s/

# See what will change
kubectl -n demo diff -f k8s/ || true

# After rollout, confirm readiness and HTTP health
kubectl -n demo get pods -l app=myapp -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'

# Curl through ClusterIP from a temporary pod
kubectl -n demo run netcheck --image=busybox:1.36 --rm -it --restart=Never -- \
  sh -c 'wget -qO- http://myapp:8080/healthz'

Local Pilot Plan

Start small, measure, and keep it easy to inspect locally before expanding.

Scope

• One service called myapp in a staging namespace demo-pilot.
• Immutable image tags using the commit SHA.

Pipeline

• Jobs: build, deploy, verify.
• Pre‑deploy: kubectl apply --dry-run=client -f k8s/ and kubectl diff -f k8s/.
• Deploy: apply manifests, set image to SHA, wait 120 s.
• Verify: curl /healthz, fetch logs, and list events.

Measurable success

• Rollout completes under 2 minutes.
• Health endpoint returns HTTP 200.
• No ImagePullBackOff or CrashLoopBackOff events in the last 50 events.

Safety and rollback

• Keep kubectl rollout undo deploy/myapp in the job on failure.
• Use a separate namespace and service account; no production secrets.

When the pilot runs cleanly for a few iterations, replicate the same pattern to a wider staging or canary environment.

Conclusion

Separate the deployment stages, provide a secure kubeconfig, and script both rollout waiting and verification to cut debugging time. Start with the pilot plan, measure the outcomes, and expand once you consistently see clean rollouts, healthy probes, and quick, one‑command rollbacks.