Kubernetes Advanced Concepts Explained with Practical Examples

Kubernetes operators frequently encounter scenarios where basic pod management and service exposure are insufficient. Production workloads demand fine-grained scheduling control, resilient storage patterns, secure network segmentation, and observable failure domains. This article covers five advanced Kubernetes concepts — pod topology spread constraints, priority classes with preemption, CSI snapshot and restore workflows, network policy egress controls, and custom metrics autoscaling — with version-scoped commands, expected output patterns, and recovery verification steps. Each section follows an observe-first approach: capture current state, apply the smallest justified change, verify the signal, and document the rollback path.

Pod Topology Spread Constraints for Failure Domain Control

Topology spread constraints distribute pods across failure domains such as zones, nodes, or custom labels. Unlike pod anti-affinity, which can leave pods unschedulable when constraints conflict, topology spread uses a scoring mechanism that favors balanced placement while remaining schedulable.

Prerequisites: Kubernetes 1.19+ (beta), 1.24+ (stable). Nodes must carry the topology label topology.kubernetes.io/zone or a custom label such as failure-domain.example.com/rack.

Observe current state:

kubectl get nodes --show-labels | grep -E 'zone|rack'
kubectl get pods -n production -l app=payment-api -o wide

Expected observation: Pods clustered on two of three available zones, leaving one zone empty.

Smallest justified change: Add a topologySpreadConstraint to the Deployment spec with maxSkew: 1, topologyKey: topology.kubernetes.io/zone, and whenUnsatisfiable: ScheduleAnyway.

spec:
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app: payment-api

Verify outcome: After rollout, kubectl get pods -n production -l app=payment-api -o jsonpath='{.items[*].spec.nodeName}' | xargs -n1 kubectl get node --show-labels should show pods spread across all three zones with a maximum difference of one pod per zone.

Recovery: Revert the Deployment to the previous revision with kubectl rollout undo deployment/payment-api -n production and confirm pod distribution returns to the prior state.

Priority Classes and Preemption for Critical Workloads

Priority classes assign scheduling precedence to pods. When cluster resources are exhausted, the scheduler preempts lower-priority pods to admit higher-priority ones. This mechanism protects control-plane components and revenue-critical services during contention.

Prerequisites: Kubernetes 1.14+ (beta), 1.19+ (stable). PriorityClass resources must exist before pods reference them.

Observe current state:

kubectl get priorityclasses
kubectl describe priorityclass system-cluster-critical
kubectl get pods -A -o custom-columns=NAME:.metadata.name,NAMESPACE:.metadata.namespace,PRIORITY:.spec.priorityClassName,NODE:.spec.nodeName

Expected observation: No custom PriorityClass for the order-processing workload; its pods run at default priority (0) and are evicted first during pressure.

Smallest justified change: Create a PriorityClass named business-critical with value: 1000000 and preemptionPolicy: PreemptLowerPriority, then patch the Deployment to reference it.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: business-critical
value: 1000000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Order processing must not be preempted by batch jobs"

kubectl patch deployment order-processor -n production -p '{"spec":{"template":{"spec":{"priorityClassName":"business-critical"}}}}'

Verify outcome: Simulate resource pressure by deploying a low-priority DaemonSet (value: 100) that requests all remaining CPU. The scheduler should preempt the DaemonSet pods to place new order-processor replicas. Confirm with kubectl get events --field-selector reason=Preempted -n production.

Recovery: Delete the PriorityClass (which fails if referenced) after removing the reference from the Deployment, then roll back the Deployment. Verify no orphaned preemption events remain.

CSI Snapshot and Restore for Stateful Workloads

Container Storage Interface (CSI) snapshots provide point-in-time copies of PersistentVolumes without application downtime. Combined with VolumeSnapshotContent and VolumeSnapshotClass, they enable backup, cloning, and disaster recovery workflows.

Prerequisites: Kubernetes 1.20+ (beta), 1.22+ (stable). CSI driver must support VOLUME_SNAPSHOT capability (e.g., AWS EBS CSI driver v1.14+, GCE PD CSI driver v1.5+, Azure Disk CSI driver v1.5+). VolumeSnapshotClass must exist in the cluster.

Observe current state:

kubectl get volumesnapshotclass
kubectl get pvc -n data-platform -l app=postgres
kubectl describe pvc postgres-data-0 -n data-platform

Expected observation: A single PVC postgres-data-0 bound to a PV provisioned by ebs.csi.aws.com, no existing snapshots.

Smallest justified change: Create a VolumeSnapshot referencing the PVC, then restore to a new PVC for a staging environment.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-snap-2024-01-15
  namespace: data-platform
spec:
  volumeSnapshotClassName: csi-aws-ebs-snapclass
  source:
    persistentVolumeClaimName: postgres-data-0

kubectl apply -f postgres-snapshot.yaml
kubectl wait --for=condition=Ready volumesnapshot/postgres-snap-2024-01-15 -n data-platform --timeout=10m

Restore:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-staging-restore
  namespace: data-platform
spec:
  storageClassName: gp3
  dataSource:
    name: postgres-snap-2024-01-15
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

Verify outcome: kubectl get pvc postgres-staging-restore -n data-platform shows Bound. Deploy a temporary PostgreSQL pod mounting this PVC and run SELECT count(*) FROM orders; to validate data integrity.

Recovery: Delete the staging PVC and VolumeSnapshot. The original PVC and PV remain untouched. Document the snapshot retention policy (e.g., daily snapshots retained for 30 days) in the runbook.

Network Policy Egress Controls for Zero-Trust Segmentation

NetworkPolicy resources restrict pod-to-pod and pod-to-external traffic. Egress rules are often overlooked but are essential for preventing data exfiltration and limiting blast radius when a workload is compromised.

Prerequisites: Kubernetes 1.8+ with a CNI plugin that enforces NetworkPolicy (Calico, Cilium, Weave Net, Antrea). kube-proxy must not be in IPVS strict-ARP mode without CNI support.

Observe current state:

kubectl get networkpolicy -n finance
kubectl exec -n finance deploy/api-gateway -- curl -s -o /dev/null -w "%{http_code}" http://external-payment-provider.example.com

Expected observation: No NetworkPolicy exists in the finance namespace; egress to any destination succeeds.

Smallest justified change: Apply a default-deny egress policy, then allow only required external endpoints by CIDR or DNS (via Cilium FQDN policy or Calico GlobalNetworkPolicy with DNS selector).

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-gateway-egress
  namespace: finance
spec:
  podSelector:
    matchLabels:
      app: api-gateway
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 203.0.113.0/24  # payment provider CIDR
      ports:
        - protocol: TCP
          port: 443
    - to:
        - namespaceSelector:
            matchLabels:
              name: monitoring
      ports:
        - protocol: TCP
          port: 9090

Verify outcome: The curl command from the observe step now returns 000 (connection refused/timeout). Allowed destinations return 200. Check Cilium/Calico logs for DENY verdicts on unexpected egress attempts.

Recovery: Delete the NetworkPolicy with kubectl delete networkpolicy api-gateway-egress -n finance. Confirm egress restores immediately. If using Cilium, verify cilium policy get shows no enforcement gaps.

Custom Metrics Autoscaling with Prometheus Adapter

Horizontal Pod Autoscaler (HPA) supports custom metrics via the Custom Metrics API. The Prometheus Adapter exposes Prometheus queries as scalable metrics, enabling autoscaling on business-level signals (queue depth, request latency, error rate) rather than CPU alone.

Prerequisites: Kubernetes 1.16+ (Custom Metrics API stable). Prometheus Operator v0.50+ with prometheus-adapter deployed. Metrics must be labeled with namespace and pod for pod-level scaling.

Observe current state:

kubectl get apiservice v1beta1.custom.metrics.k8s.io
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/shopping/pods/*/http_requests_per_second" | jq .

Expected observation: API service available. Metric returns {"items":[]} because no pods expose the metric label set.

Smallest justified change: Ensure the application emits a Prometheus counter http_requests_total with labels namespace, pod, service. Configure Prometheus Adapter rule to derive a per-second rate.

# prometheus-adapter config map snippet
rules:
  - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
    resources:
      overrides:
        namespace: {resource: "namespace"}
        pod: {resource: "pod"}
    name:
      matches: "^(.*)_total$"
      as: "${1}_per_second"
    metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

kubectl rollout restart deployment/prometheus-adapter -n monitoring

Create HPA:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-service-hpa
  namespace: shopping
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"

Verify outcome: Generate load with hey -z 5m -c 50 https://checkout.example.com. Watch kubectl get hpa checkout-service-hpa -n shopping -w — replicas should scale from 3 toward 50 as the metric exceeds 100 req/s per pod. When load stops, scale-down occurs after the stabilization window (default 5 minutes).

Recovery: Delete the HPA. Revert Prometheus Adapter config and restart. Confirm the custom metrics API no longer serves the metric.

Conclusion

Advanced Kubernetes operations require moving beyond default configurations into version-scoped, observable, and reversible patterns. Topology spread constraints eliminate silent single-zone failures. Priority classes with preemption protect revenue-critical paths during contention. CSI snapshots turn storage into a recoverable asset rather than a liability. NetworkPolicy egress rules enforce zero-trust boundaries that survive workload compromise. Custom metrics autoscaling aligns infrastructure cost with business demand signals. Each technique follows the same discipline: observe the current state with concrete commands, apply the smallest justified change, verify the expected signal, and document the rollback path before the incident occurs. Start by implementing one pattern in a staging cluster, measure the verification signals, and promote only when the recovery procedure is tested and documented.