## Intro

Deploying to Kubernetes from GitLab CI/CD becomes reliable when you understand the four moving parts: the runner, the kubeconfig, registry authentication, and rollout verification. This guide gives you concrete fixes for the most frequent failures and copy‑paste snippets you can adapt today. You’ll learn how to create a least‑privilege kubeconfig, diagnose pipeline and cluster errors, verify rollouts, and run a small pilot before scaling.

## Workflow Overview

A clear, step‑by‑step path from commit to healthy pods keeps problems isolated to a single stage:

- Build – produce a container image tagged with the commit SHA.

- Push – authenticate to the registry and push the image.

- Configure – supply kubectl and a kubeconfig that grants only the needed permissions.

- Apply – run kubectl apply or helm upgrade --install .

- Wait – watch the rollout status with a timeout.

- Verify – run a quick smoke test against the Service or Ingress.

- Roll back – revert to the last good ReplicaSet if verification fails.

Keep each step in its own job so you can see exactly where a failure occurs.

## Setup: runners and kubeconfig

### Service account and RBAC (namespace‑scoped)

Create a dedicated service account with the minimum rights required in the target namespace:

apiVersion: v1
kind: Namespace
metadata:
 name: demo
---
apiVersion: v1
kind: ServiceAccount
metadata:
 name: ci-deployer
 namespace: demo
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: ci-deployer
 namespace: demo
rules:
- apiGroups: ["", "apps", "extensions
esources: ["pods", "deployments", "replicasets", "services", "configmaps", "secrets
 verbs: ["get", "list", "watch", "create", "update", "patch", "delete
d---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: ci-deployer
 namespace: demo
subjects:
- kind: ServiceAccount
 name: ci-deployer
 namespace: demo
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: ci-deployer 
 Generate a short‑lived token (Kubernetes 1.24+):

kubectl -n demo create token ci-deployer > ci-deployer.token 
 Build a kubeconfig that points to your API server and includes the CA data, then store the full YAML as a protected, masked CI variable named KUBE_CONFIG_B64 (base64‑encoded) or KUBE_CONFIG_YAML (multiline).

### Writing kubeconfig in the job

# .gitlab-ci.yml (kubeconfig snippet)
before_script:
 - mkdir -p ~/.kube
 - |
 if [ -n "$KUBE_CONFIG_B64" ]; then
 echo "$KUBE_CONFIG_B64" | base64 -d > ~/.kube/config
 else
 echo "$KUBE_CONFIG_YAML" > ~/.kube/config
 fi
 - chmod 600 ~/.kube/config 

### Runner image choice

 Use a job image that already contains kubectl (for example bitnami/kubectl:latest ) or install kubectl at runtime. If you build images with Docker‑in‑Docker, the runner must run in privileged mode. When privileged runners are not an option, switch to a rootless builder such as Kaniko or BuildKit.

## Reference pipeline with build, push, and deploy

stages: [build, deploy]

variables:
 IMAGE: "$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA"
 KUBE_NAMESPACE: "demo"

# Build and push image using Docker-in-Docker (requires privileged runner)
build:
 stage: build
 image: docker:25
 services:
 - name: docker:25-dind
 command: ["--mtu=1460\
 variables:
 DOCKER_TLS_CERTDIR: /certs
 script:
 - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"
 - docker build -t "$IMAGE" .
 - docker push "$IMAGE"
 rules:
 - if: "$CI_COMMIT_BRANCH"

# Deploy using kubectl apply and wait for rollout
deploy:
 stage: deploy
 image: bitnami/kubectl:latest
 before_script:
 - mkdir -p ~/.kube
 - if [ -n "$KUBE_CONFIG_B64" ]; then echo "$KUBE_CONFIG_B64" | base64 -d > ~/.kube/config; else echo "$KUBE_CONFIG_YAML" > ~/.kube/config; fi
 - chmod 600 ~/.kube/config
 script:
 # Ensure namespace exists
 - kubectl get ns "$KUBE_NAMESPACE" || kubectl create ns "$KUBE_NAMESPACE"

 # Create or update image pull secret for the namespace (once per namespace)
 - |
 kubectl -n "$KUBE_NAMESPACE" create secret docker-registry gitlab-regcred \
 --docker-server="$CI_REGISTRY" \
 --docker-username="$CI_REGISTRY_USER" \
 --docker-password="$CI_REGISTRY_PASSWORD" \
 --docker-email="devnull@example.com" \
 --dry-run=client -o yaml | kubectl apply -f -

 # Apply manifests (deployment uses imagePullSecrets)
 - kubectl -n "$KUBE_NAMESPACE" apply -f k8s/

 # Set the container image to the commit SHA tag (optional patch)
 - kubectl -n "$KUBE_NAMESPACE" set image deploy/myapp myapp-container="$IMAGE" --record=true || true

 # Wait for rollout with timeout
 - kubectl -n "$KUBE_NAMESPACE" rollout status deploy/myapp --timeout=120s

 # Simple smoke check against ClusterIP via busybox curl (optional)
 - kubectl -n "$KUBE_NAMESPACE" run tmp-curl --image=busybox:1.36 --restart=Never --rm -it -- curl -sS myapp:8080/healthz
 environment:
 name: demo/$CI_COMMIT_REF_NAME
 url: https://demo.example.com
 rules:
 - if: "$CI_COMMIT_BRANCH == \"main\"" 
 Example Deployment manifest ( k8s/deploy.yaml ):

apiVersion: apps/v1
kind: Deployment
metadata:
 name: myapp
 labels:
 app: myapp
spec:
 replicas: 2
 selector:
 matchLabels:
 app: myapp
 template:
 metadata:
 labels:
 app: myapp
 spec:
 serviceAccountName: ci-deployer
 imagePullSecrets:
 - name: gitlab-regcred
 containers:
 - name: myapp-container
 image: registry.example.com/group/project:CHANGE_ME
 imagePullPolicy: Always
 ports:
 - containerPort: 8080
 readinessProbe:
 httpGet:
 path: /healthz
 port: 8080
 initialDelaySeconds: 5
 periodSeconds: 5
 livenessProbe:
 httpGet:
 path: /healthz
 port: 8080
 initialDelaySeconds: 10
 periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
 name: myapp
spec:
 selector:
 app: myapp
 ports:
 - port: 8080
 targetPort: 8080 

## Troubleshooting: pipeline failures

 Symptom Likely cause Quick fix 
 Job stuck in pending No runner matches the job tags or all runners are busy Add tags that match a registered runner; verify runner status in project settings 
 docker build fails or cannot connect to Docker daemon DinD not enabled or runner not privileged Enable privileged mode for the runner or switch to Kaniko; ensure DOCKER_HOST and DOCKER_TLS_CERTDIR are set when using DinD 
 kubectl: command not found Job image lacks kubectl Use an image with kubectl (e.g., bitnami/kubectl ) or install it in the job 
 The request could not be authenticated / Forbidden Invalid/expired kubeconfig token or insufficient RBAC Refresh the service‑account token, verify the API server URL and CA in the kubeconfig, and grant the required verbs on the target namespace resources 
 apply fails with no matches for kind or invalid Wrong apiVersion or schema mismatch Validate locally with kubectl apply --dry-run=client -f k8s/ and pin correct apiVersion values 
 Namespace not found Deploying into a namespace that does not exist Create the namespace before applying resources ( kubectl create ns <name> ) 

## Troubleshooting: image pull errors

 Pod shows ErrImagePull or ImagePullBackOff :

- Wrong image tag

- Check the exact tag in the Deployment:

kubectl -n demo get deploy myapp -o jsonpath='{.spec.template.spec.containers[0].image}' 

- Prefer immutable tags such as $CI_COMMIT_SHORT_SHA .

- Registry authentication missing

- Ensure a docker-registry secret exists in the same namespace and is referenced by imagePullSecrets or the service account.

- Recreate the secret if credentials changed:

 kubectl -n demo delete secret gitlab-regcred --ignore-not-found
 kubectl -n demo create secret docker-registry gitlab-regcred \
 --docker-server="$CI_REGISTRY" \
 --docker-username="$CI_REGISTRY_USER" \
 --docker-password="$CI_REGISTRY_PASSWORD" \
 --docker-email="devnull@example.com" 

- Policy and caching surprises

- If you use :latest , set imagePullPolicy: Always to avoid stale images.

- Better: pin to the commit SHA and keep imagePullPolicy: IfNotPresent .

- Secrets are namespace‑scoped; create the same pull secret in each target namespace.

 To confirm the fix, delete the failing pod and let the ReplicaSet recreate it, or trigger a rollout restart:

kubectl -n demo rollout restart deploy/myapp 

## Troubleshooting: rollout checks and debugging

 Automate the wait and enrich failure logs:

kubectl -n demo rollout status deploy/myapp --timeout=120s || {
 echo "Rollout did not complete in time"
 kubectl -n demo get pods -o wide
 kubectl -n demo describe deploy/myapp
 kubectl -n demo get events --sort-by=.lastTimestamp | tail -n 50
 exit 1
} 
 If rollout fails or pods crash:

- Describe and events – kubectl -n demo describe pod <name> shows image‑pull, scheduling, and probe issues. kubectl -n demo get events --sort-by=.lastTimestamp surfaces recent errors.

- Logs – kubectl -n demo logs deploy/myapp --all-containers=true --tail=200 for application errors.

- Probes and resources – Readiness failures block rollout. Verify the probe path and port; increase initialDelaySeconds if the app starts slowly. Check CPU/memory requests and limits; OOMKilled means the memory limit is too low.

- Services and Ingress – Ensure Service selectors match pod labels exactly. Confirm targetPort matches the container port. For Ingress, validate host, TLS, and that the Service backends are healthy.

- ConfigMaps and Secrets – Missing keys or wrong mounts often cause CrashLoopBackOff . Compare the running pod spec to the manifest: kubectl -n demo get pod <name> -o yaml .

Fast rollback:

kubectl -n demo rollout undo deploy/myapp 

## Scripted safety checks in CI

 Add preflight and post‑deploy checks to reduce guesswork:

# Validate manifests before apply
kubectl -n demo apply --dry-run=client -f k8s/

# See what will change
kubectl -n demo diff -f k8s/ || true

# After rollout, confirm readiness and HTTP health
kubectl -n demo get pods -l app=myapp -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'

# Curl through ClusterIP from a temporary pod
kubectl -n demo run netcheck --image=busybox:1.36 --rm -it --restart=Never -- \
 sh -c 'wget -qO- http://myapp:8080/healthz' 

## Local Pilot Plan

 Start small, measure, and keep it easy to inspect locally before expanding.

Scope

- One service called myapp in a staging namespace demo-pilot .

- Immutable image tags using the commit SHA.

Pipeline

- Jobs: build , deploy , verify .

- Pre‑deploy: kubectl apply --dry-run=client -f k8s/ and kubectl diff -f k8s/ .

- Deploy: apply manifests, set image to SHA, wait 120 s.

- Verify: curl /healthz , fetch logs, and list events.

Measurable success

- Rollout completes under 2 minutes.

- Health endpoint returns HTTP 200.

- No ImagePullBackOff or CrashLoopBackOff events in the last 50 events.

Safety and rollback

- Keep kubectl rollout undo deploy/myapp in the job on failure.

- Use a separate namespace and service account; no production secrets.

When the pilot runs cleanly for a few iterations, replicate the same pattern to a wider staging or canary environment.

## Conclusion

Separate the deployment stages, provide a secure kubeconfig, and script both rollout waiting and verification to cut debugging time. Start with the pilot plan, measure the outcomes, and expand once you consistently see clean rollouts, healthy probes, and quick, one‑command rollbacks.