GitLab CI/CD Kubernetes deployment troubleshooting with practical examples: practical implementation guide

Intro

Deploying to Kubernetes from GitLab CI/CD is straightforward once the moving parts are clear: the runner, kubeconfig, registry auth, and rollout checks. This guide focuses on practical fixes for common problems and includes copy-paste examples you can adapt today. You will learn how to set up a safe kubeconfig, diagnose pipeline and cluster errors, verify rollouts, and run a small pilot before scaling up.

Workflow Overview

A simple, inspectable path from commit to healthy pods keeps issues contained to a single step:

1. Build: create a container image tagged with the commit SHA.
2. Push: authenticate to the registry and push the image.
3. Configure: provide kubectl and a kubeconfig with least-privilege access.
4. Apply: apply manifests or run Helm upgrade --install.
5. Wait: check rollout status with a timeout.
6. Verify: run smoke checks against Service/Ingress.
7. Roll back: undo to the last good ReplicaSet if verification fails.

Keep each step as a separate job so you can see exactly where failures happen.

Setup: runners and kubeconfig

Service account and RBAC (namespace-scoped)

Use a dedicated service account with least privilege in your target namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ci-deployer
  namespace: demo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: ci-deployer
  namespace: demo
rules:
- apiGroups: ["", "apps", "extensions"]
  resources: ["pods", "deployments", "replicasets", "services", "configmaps", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: ci-deployer
  namespace: demo
subjects:
- kind: ServiceAccount
  name: ci-deployer
  namespace: demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: ci-deployer

Create a short-lived token for the service account (Kubernetes 1.24+):

kubectl -n demo create token ci-deployer > ci-deployer.token

Build a kubeconfig that uses your API server and CA data, then store the full YAML as a protected, masked CI variable named KUBE_CONFIG_B64 (base64-encoded) or KUBE_CONFIG_YAML (multiline).

Writing kubeconfig in the job

# .gitlab-ci.yml (kubeconfig snippet)
before_script:
  - mkdir -p ~/.kube
  - |
    if [ -n "$KUBE_CONFIG_B64" ]; then
      echo "$KUBE_CONFIG_B64" | base64 -d > ~/.kube/config
    else
      echo "$KUBE_CONFIG_YAML" > ~/.kube/config
    fi
  - chmod 600 ~/.kube/config

Runner image choice

Use a job image that includes kubectl (for example, bitnami/kubectl: latest) or install kubectl at runtime. If you build images in CI with docker-in-docker, the runner must be privileged. If that is not an option, use a builder like Kaniko or BuildKit-in-rootless mode.

Reference pipeline with build, push, and deploy

stages: [build, deploy]

variables:
  IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
  KUBE_NAMESPACE: "demo"

# Build and push image using Docker-in-Docker (requires privileged runner)
build:
  stage: build
  image: docker:25
  services:
    - name: docker:25-dind
      command: ["--mtu=1460"]
  variables:
    DOCKER_TLS_CERTDIR: "/certs"
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
    - docker build -t "$IMAGE" .
    - docker push "$IMAGE"
  rules:
    - if: "$CI_COMMIT_BRANCH"

# Deploy using kubectl apply and wait for rollout
deploy:
  stage: deploy
  image: bitnami/kubectl: latest
  before_script:
    - mkdir -p ~/.kube
    - if [ -n "$KUBE_CONFIG_B64" ]; then echo "$KUBE_CONFIG_B64" | base64 -d > ~/.kube/config; else echo "$KUBE_CONFIG_YAML" > ~/.kube/config; fi
    - chmod 600 ~/.kube/config
  script:
    # Ensure namespace exists
    - kubectl get ns "$KUBE_NAMESPACE" || kubectl create ns "$KUBE_NAMESPACE"

    # Create or update image pull secret for the namespace (once per namespace)
    - |
      kubectl -n "$KUBE_NAMESPACE" create secret docker-registry gitlab-regcred \
        --docker-server="$CI_REGISTRY" \
        --docker-username="$CI_REGISTRY_USER" \
        --docker-password="$CI_REGISTRY_PASSWORD" \
        --docker-email="[email protected]" \
        --dry-run=client -o yaml | kubectl apply -f -

    # Apply manifests (deployment uses imagePullSecrets)
    - kubectl -n "$KUBE_NAMESPACE" apply -f k8s/

    # Set the container image to the commit SHA tag (optional patch)
    - kubectl -n "$KUBE_NAMESPACE" set image deploy/myapp myapp-container="$IMAGE" --record=true || true

    # Wait for rollout with timeout
    - kubectl -n "$KUBE_NAMESPACE" rollout status deploy/myapp --timeout=120s

    # Simple smoke check against ClusterIP via busybox curl (optional)
    - kubectl -n "$KUBE_NAMESPACE" run tmp-curl --image=busybox:1.36 --restart=Never --rm -it -- curl -sS myapp:8080/healthz
  environment:
    name: demo/$CI_COMMIT_REF_NAME
    url: https://demo.example.com
  rules:
    - if: "$CI_COMMIT_BRANCH == \"main\""

Example Deployment manifest (k8s/deploy.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      serviceAccountName: ci-deployer
      imagePullSecrets:
        - name: gitlab-regcred
      containers:
        - name: myapp-container
          image: registry.example.com/group/project: CHANGE_ME
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 5
            periodSeconds: 5
          livenessProbe:
            httpGet: { path: /healthz, port: 8080 }
            initialDelaySeconds: 10
            periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: myapp
spec:
  selector:
    app: myapp
  ports:
    - port: 8080
      targetPort: 8080

Troubleshooting: pipeline failures

Symptoms and fast checks:

• Job stuck in pending
• Cause: no runner matches tags or no free runners.
• Fix: add tags to the job that match the registered runner. Verify runner status in project settings.

• docker build fails or cannot connect to docker
• Cause: dind not enabled or runner is not privileged.
• Fix: enable privileged mode for the runner or switch to Kaniko. Ensure DOCKER_HOST and DOCKER_TLS_CERTDIR are set when using dind.

• kubectl: command not found
• Cause: job image does not include kubectl.
• Fix: use a kubectl-capable image (for example bitnami/kubectl) or install kubectl in the job.

• The request could not be authenticated or forbidden
• Cause: kubeconfig invalid, expired token, or insufficient RBAC.
• Fix: refresh the service account token, verify the API server URL and CA in kubeconfig, and grant required verbs on resources in the target namespace.

• apply fails with no matches for kind or invalid
• Cause: wrong apiVersion or schema.
• Fix: validate manifests locally: kubectl apply --dry-run=client -f k8s/. Pin valid apiVersions.

• Namespace not found
• Cause: deploying into a namespace that does not exist.
• Fix: create the namespace before applying resources.

Troubleshooting: image pull errors

Pod shows ErrImagePull or ImagePullBackOff:

1. Wrong image tag

• Check the exact tag in the Deployment: kubectl -n demo get deploy myapp -o jsonpath='{.spec.template.spec.containers[0].image}'.
• Prefer immutable tags: $CI_COMMIT_SHORT_SHA.

1. Registry auth missing

• Ensure a docker-registry Secret exists in the same namespace and is referenced by imagePullSecrets or the ServiceAccount.
• Recreate the secret if credentials changed:

kubectl -n demo delete secret gitlab-regcred --ignore-not-found
kubectl -n demo create secret docker-registry gitlab-regcred \
  --docker-server="$CI_REGISTRY" \
  --docker-username="$CI_REGISTRY_USER" \
  --docker-password="$CI_REGISTRY_PASSWORD" \
  --docker-email="[email protected]"

1. Policy and caching surprises

• If using :latest, set imagePullPolicy: Always to avoid stale images.
• Better: pin to the commit SHA and keep imagePullPolicy: IfNotPresent.

1. Cross-namespace pulls

• Secrets are namespace-scoped. Create the same pull secret in each target namespace.

To confirm the fix, delete the failing Pod and let the ReplicaSet recreate it, or trigger a rollout with kubectl rollout restart deploy/myapp.

Troubleshooting: rollout checks and debugging

Automate the wait and enrich failure logs:

# Wait with timeout
kubectl -n demo rollout status deploy/myapp --timeout=120s || {
  echo "Rollout did not complete in time";
  kubectl -n demo get pods -o wide;
  kubectl -n demo describe deploy/myapp;
  kubectl -n demo get events --sort-by=.lastTimestamp | tail -n 50;
  exit 1;
}

If rollout fails or pods crash:

• Describe and events
• kubectl -n demo describe pod <name> shows image pull, scheduling, and probe issues.
• kubectl -n demo get events --sort-by=.lastTimestamp surfaces recent errors.

• Logs
• kubectl -n demo logs deploy/myapp --all-containers=true --tail=200 for app errors.

• Probes and resources
• Readiness failing blocks rollout. Verify the path and port. If your app is slow to start, increase initialDelaySeconds.
• Check CPU/memory requests and limits. OOMKilled indicates too little memory.

• Services and Ingress
• Ensure Service selectors match Pod labels exactly.
• Confirm the targetPort matches the containerPort.
• For Ingress, validate host, TLS, and that the Service backends are healthy.

• ConfigMaps and Secrets
• Missing keys or wrong mounts often cause CrashLoopBackOff. Compare running Pod spec to manifests: kubectl -n demo get pod <name> -o yaml.

Fast rollback:

kubectl -n demo rollout undo deploy/myapp

Scripted safety checks in CI

Add preflight and post-deploy checks to reduce guesswork:

# Validate manifests before apply
kubectl -n demo apply --dry-run=client -f k8s/

# See what will change
kubectl -n demo diff -f k8s/ || true

# After rollout, confirm readiness and HTTP health
kubectl -n demo get pods -l app=myapp -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'

# Curl through ClusterIP from a temporary Pod
kubectl -n demo run netcheck --image=busybox:1.36 --rm -it --restart=Never -- \
  sh -c 'wget -qO- http://myapp:8080/healthz'

Local Pilot Plan

Start small, measure, and keep it easy to inspect locally before deploying.

Scope

• One service called myapp in a staging namespace demo-pilot.
• Immutable image tags using the commit SHA.

Pipeline

• Jobs: build, deploy, verify.
• Pre-deploy: kubectl apply --dry-run=client -f k8s/ and kubectl diff -f k8s/.
• Deploy: apply manifests, set image to SHA, wait 120s.
• Verify: curl /healthz, fetch logs, and list events.

Measurable success

• Rollout completes under 2 minutes.
• Health endpoint returns 200.
• No ImagePullBackOff or CrashLoopBackOff events in the last 50 events.

Safety and rollback

• Keep kubectl rollout undo deploy/myapp in the job on failure.
• Use a separate namespace and ServiceAccount. No production secrets.

When stable for a few runs, replicate the same pattern to a wider staging or canary environment.

Conclusion

Separate the deployment stages, provide a secure kubeconfig, and script both rollout and verification to shorten debugging time. Start with the pilot plan, measure outcomes, and expand once you consistently see clean rollouts, healthy probes, and quick, one-command rollbacks.