Kubernetes Topology Manager Upgrades and Migrations: A Practical Implementation Guide

Intro

Upgrading or migrating the Kubernetes Topology Manager requires more than changing a version string. It is an operational change that can alter how the kubelet aligns CPU, memory, and device resources across NUMA nodes. A careless upgrade can cause pod admission failures, unexpected resource alignment, or even node instability. This guide provides a practical, hands-on approach for operators, DevOps engineers, and platform teams who need to plan, execute, verify, and recover from Topology Manager changes.

We will walk through a complete workflow: inventorying the current environment, designing a safe configuration path, validating changes before rollout, diagnosing failures, and rolling back if needed. Every section includes concrete commands, realistic expected outputs, and decision criteria. The examples assume a Kubernetes cluster running version 1.27 or later, but the principles apply to any supported release.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before touching any configuration, you must know exactly what you are running. This section covers collecting the necessary information about your Kubernetes version, node topology, current Topology Manager configuration, and relevant components.

1. Identify Kubernetes and kubelet versions

Run the following on a control plane node:

kubectl version --short

Since Kubernetes 1.26, the --short flag is deprecated but still widely used. A typical output looks like:

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

On each worker node, check the kubelet version:

kubelet --version

Example output:

Kubernetes v1.27.3

Record both the control plane and kubelet versions. Topology Manager behavior can differ between kubelet versions, even if the control plane is the same.

2. Check current Topology Manager configuration

The kubelet configuration file usually resides at /var/lib/kubelet/config.yaml. Inspect it for topology-related fields:

sudo cat /var/lib/kubelet/config.yaml | grep -A 20 topologyManager

If the Topology Manager is not configured, you will see no output or the following defaults:

topologyManagerPolicy: none

Other possible values are:

• none: topology manager disabled (default)
• best-effort: align resources when possible, but never reject pods
• restricted: align resources strictly, reject pods that cannot be aligned
• single-numa-node: same as restricted, but also require all resources to fit on a single NUMA node (deprecated in 1.27, use restricted with topologyManagerScope: pod or container)

Also check the topologyManagerScope:

topologyManagerScope: container  # or pod

And the CPU Manager policy, because Topology Manager interacts with it:

cpuManagerPolicy: static

If you are using a kubelet configuration file, verify that the kubelet is actually using it. Look at the kubelet systemd unit:

systemctl cat kubelet

Find the --config flag. It should point to your file. If not, the kubelet may be using command-line flags, which take precedence. In that case, check the command line:

ps aux | grep kubelet

Look for flags like --topology-manager-policy and --cpu-manager-policy.

3. Inspect node NUMA topology

You need to know the NUMA layout of your nodes. Use lscpu on a node:

lscpu | grep -E 'NUMA|CPU\(s\)|On-line'

Example output on a two-socket server:

CPU(s):              32
On-line CPU(s) list: 0-31
NUMA node(s):        2
NUMA node0 CPU(s):   0-15
NUMA node1 CPU(s):   16-31

This tells you that the node has two NUMA nodes, each with 16 CPUs.

4. Check device plugins and resources

If you use devices like GPUs or SR-IOV NICs, list the extended resources on nodes:

kubectl describe nodes | grep -A 5 'Capacity:'

Look for entries like nvidia.com/gpu: 2 or intel.com/sriov: 4. These devices are managed by device plugins, and Topology Manager will consider their NUMA affinity.

5. Verify current pod scheduling behavior

Before changing anything, observe how current pods are placed. For a running pod, you can see its assigned CPUs and NUMA nodes:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].resources}'

To see actual CPU assignment on the node, you can use crictl (if configured) or examine the cgroup:

sudo crictl inspect <container-id> | grep -A 5 cpuset

However, this is often not easily accessible. A simpler method is to check the kubelet logs for topology manager decisions:

sudo journalctl -u kubelet | grep -i topology

Look for lines indicating alignment hints or failures.

Practical checklist for inventory

• [ ] Record Kubernetes server and kubelet versions.
• [ ] Document current topologyManagerPolicy and topologyManagerScope.
• [ ] Document current cpuManagerPolicy and any related flags.
• [ ] Capture NUMA topology for each node type.
• [ ] List extended resources and device plugins.
• [ ] Baseline current pod scheduling behavior (select a few representative pods).
• [ ] Store this inventory in a versioned document or Git repository.

Safe Configuration Path

Once you have a full inventory, plan the configuration changes carefully. The key principle is to make one change at a time and verify its effect before proceeding.

1. Decide on target configuration

Consider what you want to achieve:

• Migrating from none to best-effort: minimal risk, improves alignment when possible without rejecting pods.
• Migrating from none or best-effort to restricted: higher risk; pods that cannot be aligned will be rejected.
• Changing topologyManagerScope from container to pod: may increase alignment strictness.
• Upgrading Kubernetes: may bring new Topology Manager features or deprecated old ones.

For a typical first step, enabling best-effort is a safe improvement. Example kubelet configuration snippet:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
topologyManagerPolicy: best-effort
topologyManagerScope: container
cpuManagerPolicy: static
cpuManagerReconcilePeriod: 10s

2. Choose a test node

Never roll out changes to all nodes at once. Pick one worker node that is not running critical workloads or can tolerate disruption. Drain the node before making changes:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

This will evict all pods from the node. The --delete-emptydir-data flag is necessary if any pods use emptyDir volumes; otherwise, the drain will hang.

3. Update kubelet configuration

On the test node, edit the kubelet configuration file (e.g., /var/lib/kubelet/config.yaml). Make a backup first:

sudo cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak

Then apply your changes. For example, to enable best-effort:

sudo sed -i 's/topologyManagerPolicy: none/topologyManagerPolicy: best-effort/' /var/lib/kubelet/config.yaml

If the line does not exist, add it under the appropriate section. Ensure YAML indentation is correct.

4. Restart kubelet

Restart the kubelet to apply the new configuration:

sudo systemctl restart kubelet

Check that it is running:

sudo systemctl status kubelet

Look for any error messages in the journal:

sudo journalctl -u kubelet --since "5 minutes ago" | grep -i error

5. Uncordon the node and test scheduling

Allow pods to be scheduled again:

kubectl uncordon <node-name>

Then deploy a simple test pod that requests multiple resources, such as CPU and memory, to see how Topology Manager aligns them. For example:

apiVersion: v1
kind: Pod
metadata:
  name: topology-test
spec:
  containers:
  - name: test
    image: nginx
    resources:
      requests:
        cpu: "2"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "1Gi"

Apply and check its scheduling:

kubectl apply -f topology-test.yaml
kubectl get pods -o wide

If the pod is scheduled, check the node's kubelet logs for topology manager decisions:

sudo journalctl -u kubelet | grep -i topology | tail -20

Look for messages indicating alignment hints or successful allocation.

6. Monitor and compare

Keep the test pod running for a while and monitor for any unexpected behavior. Then delete it:

kubectl delete pod topology-test

Compare the node's behavior before and after the change. If all is well, proceed to the next node, one at a time.

Verification and Diagnostics

After applying changes, you need to verify that the Topology Manager is enforcing the intended policy and that pods are being scheduled as expected. This section provides commands and diagnostic techniques.

1. Verify active configuration

On the node, check the kubelet's effective configuration:

sudo kubectl get --raw /api/v1/nodes/<node-name>/proxy/configz | jq '.kubeletconfig.topologyManagerPolicy'

This queries the kubelet's configz endpoint (requires the API server to have proxy access). The output should show your configured policy, e.g., "best-effort".

If configz is not available, check the running kubelet process:

ps aux | grep kubelet | grep -o 'topology-manager-policy=[^ ]*'

If using a config file, you can also verify the file contents:

sudo grep topologyManagerPolicy /var/lib/kubelet/config.yaml

2. Test with a pod that requires alignment

Deploy a pod with Guaranteed QoS (requests equal limits) and multiple resources:

apiVersion: v1
kind: Pod
metadata:
  name: strict-alignment-test
spec:
  containers:
  - name: test
    image: nginx
    resources:
      requests:
        cpu: "4"
        memory: "4Gi"
        ephemeral-storage: "1Gi"
      limits:
        cpu: "4"
        memory: "4Gi"
        ephemeral-storage: "1Gi"

This pod requests a specific amount of CPU and memory that must be aligned. Apply and check if it is scheduled:

kubectl apply -f strict-alignment-test.yaml
kubectl get pods -o wide

If the policy is restricted and the node cannot satisfy the alignment, the pod will remain in Pending state. Check the pod events for a detailed message:

kubectl describe pod strict-alignment-test

Look for an event like:

Warning  FailedScheduling  10s   default-scheduler  0/3 nodes are available: 3 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) didn't match Pod's node affinity/selector.

Or, if the scheduler allowed it but the kubelet rejected it:

Warning  Failed  12s  kubelet  admission failed: topology manager policy "restricted" cannot allocate resources

3. Inspect kubelet logs for topology decisions

On the node, use journalctl to see detailed logs:

sudo journalctl -u kubelet --since "10 minutes ago" | grep -i topology

For a successful allocation, you might see:

I0523 10:15:32.123456   12345 topology_manager.go:215] [topologymanager] Topology Manager policy best-effort, hints: {01 true}

For a failure under restricted:

E0523 10:20:11.654321   12345 topology_manager.go:187] [topologymanager] Topology Manager policy restricted, unable to find aligned resources for container test

4. Use Node Resource Topology (if available)

Kubernetes 1.27+ may have the Node Resource Topology API enabled. You can query it to see resource availability per NUMA node:

kubectl get noderesourcetopologies

Example output:

NAME          AGE
node-1        5d
node-2        5d

Describe one:

kubectl describe noderesourcetopologies node-1

Look for topologyPolicies and zones sections to understand the NUMA layout and resources.

If this API is not enabled, you can still inspect the node's resource allocation via the kubelet's /metrics endpoint for container_resources_cpu_usage but that is less topology-specific.

5. Diagnostic checklist

• Confirm the kubelet is using the intended configuration.
• Deploy a test pod that requires alignment and observe its scheduling.
• Check pod events and kubelet logs for topology-related messages.
• If the pod is rejected, understand why (resource distribution, policy too strict).
• If the pod is accepted, verify the actual CPU/NUMA assignment (if possible via cgroup cpuset).
• Compare behavior across nodes if you have multiple node types.

Failure Modes and Recovery

No matter how careful you are, things can go wrong. This section covers common failure scenarios and how to recover from them.

1. Kubelet fails to start after configuration change

Symptom: After editing the config file and restarting kubelet, the node becomes NotReady.

Diagnosis: Check kubelet status and logs:

sudo systemctl status kubelet
sudo journalctl -u kubelet --since "5 minutes ago"

Possible errors include malformed YAML or invalid policy value.

Recovery: Restore the backup configuration:

sudo cp /var/lib/kubelet/config.yaml.bak /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet

Wait for the node to become Ready:

kubectl get nodes

2. Pods fail to schedule or are evicted

Symptom: After enabling restricted or changing scope, some pods go to Pending or get evicted.

Diagnosis: Check pod events:

kubectl describe pod <pod-name>

Look for messages about topology manager mismatch or insufficient resources.

Recovery: If the pods are critical, revert the topology manager policy to best-effort or none on the affected node(s), following the same drain-cordon procedure. Alternatively, adjust the pod's resource requests to be alignable.

3. Performance degradation due to suboptimal alignment

Symptom: Applications run slower after the change, possibly because they are now pinned to a single NUMA node and cannot use remote memory.

Diagnosis: Compare application metrics before and after. Check if the pod is restricted to a single NUMA node (using cgroup cpuset).

Recovery: Consider using best-effort instead of restricted to allow more flexibility. Or adjust the pod's resources to fit within a NUMA node if you want strict alignment.

4. Device plugin compatibility issues

Symptom: Pods requesting GPU or other devices fail to schedule after Topology Manager change.

Diagnosis: Check device plugin logs and kubelet logs for errors. The device plugin may not support the topology hints properly.

Recovery: Update the device plugin to a version that supports Topology Manager. Or disable Topology Manager for pods that use those devices by using a different namespace or annotation (if supported).

5. Inconsistent behavior across nodes

Symptom: Some nodes accept pods while others reject them, even though they have similar specs.

Diagnosis: Check NUMA topology differences. Some nodes may have different number of NUMA nodes or varied resource distribution.

Recovery: Standardize node hardware, or adjust the policy per node type. You can use kubelet configuration overrides per node group (e.g., via node labels and separate config files).

General recovery strategies

• Always keep a backup of the kubelet configuration before changes.
• Make changes one node at a time to limit impact.
• Have a method to drain and cordon nodes quickly.
• Document the exact revert procedure.
• Test the rollback on a non-production node first, if possible.

Common Pitfalls and How to Avoid Them

This section highlights frequent mistakes and provides guidance to prevent or recover from them.

1. Not checking hardware NUMA topology before enabling strict policies

Pitfall: Enabling restricted or single-numa-node on nodes with multiple NUMA nodes without understanding resource distribution can lead to many pod rejections.

How to avoid: Use lscpu to map NUMA nodes and CPUs. Test with best-effort first and monitor alignment success.

Recovery: Revert to best-effort if too many pods fail.

2. Ignoring the kubelet configuration via command-line flags

Pitfall: Editing the config file but not realizing that the kubelet is using command-line flags that override the file.

How to avoid: Check ps aux | grep kubelet and the systemd unit for --config vs. individual flags. Use one method consistently.

Recovery: Update the flags or remove them and rely on the config file.

3. Applying changes to all nodes at once

Pitfall: Rolling out a topology manager change cluster-wide without a canary can cause widespread scheduling failures.

How to avoid: Use a phased rollout: one node at a time, monitor, then proceed. Consider using node pools or machine deployments to manage different configurations.

Recovery: If you already rolled out to many nodes, drain and revert each node quickly, or use a configuration management tool to push the old config.

4. Overlooking the impact on Guaranteed pods

Pitfall: Guaranteed QoS pods (requests equal limits) are always considered for strict alignment. If you enable restricted, all such pods must be alignable.

How to avoid: Before enabling, review existing Guaranteed pods and their resource requests. Determine if they can fit within NUMA constraints.

Recovery: Adjust pod resources or use best-effort if alignment is not critical.

5. Not accounting for CPU Manager policy interaction

Pitfall: Topology Manager works closely with CPU Manager. If CPU Manager policy is none, Topology Manager has limited effect for CPU.

How to avoid: Decide on CPU Manager policy (static is recommended for strict alignment) and configure it alongside Topology Manager.

Recovery: Ensure both policies are consistent.

6. Forgetting to check extended resources and device plugins

Pitfall: Devices like GPUs may have their own topology hints. If the device plugin is outdated or misconfigured, Topology Manager may not get correct hints.

How to avoid: Update device plugins to versions that support Topology Manager. Test with dummy devices if possible.

Recovery: Disable Topology Manager for device-using pods or update the plugin.

Operations Checklist

Use this checklist to ensure a smooth upgrade or migration. Each item includes an owner (a single accountable person) and a review frequency.

Conclusion

Upgrading or migrating the Kubernetes Topology Manager is a delicate operation that requires careful planning, incremental execution, and thorough verification. By following the procedures in this guide - from initial inventory to safe configuration, validation, diagnostics, and recovery - you can minimize risk and achieve better resource alignment in your cluster.

Remember that every cluster is different, and what works in one environment may not work in another. Always test in a staging or canary environment first, and have a solid rollback plan. Keep your kubelet configurations under version control, and document every change.

As a next step, choose one low-risk verification (e.g., enabling best-effort on a single node) and run through the checklist. Record current state, execute the change, compare expected vs. actual behavior, and review dependencies such as CPU Manager, device plugins, and scheduling policies. With a methodical approach, you can safely leverage Topology Manager to improve workload performance and resource utilization.