## Intro Upgrading Proxmox Virtual Environment (VE) and migrating workloads between nodes are routine but high-stakes operations. A mistake can take down multiple virtual machines, corrupt storage, or leave a cluster in a split-brain state. This guide provides a practical, command-driven approach for developers, DevOps consultants, and technical startup teams who need to perform Proxmox upgrades and migrations safely. We focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. Every major step includes concrete commands, expected output, and failure signals. The guide covers Proxmox VE 8.x, the current stable release as of this writing, but the principles apply to earlier versions. ## Version and Environment Inventory Before touching anything, you must know exactly what you are running. Start by gathering version information and deployment topology using read-only commands. This establishes a baseline and helps identify compatibility issues. ### Check Proxmox VE Version Run the following on each node: pveversion Expected output example: pve-manager/8.1.4/6b6e2a6d (running kernel: 6.5.11-7-pve) This shows the Proxmox VE manager version, the kernel, and a build hash. Note the kernel version because some upgrades require a reboot into a new kernel. ### Check Cluster Status If you are in a cluster, ensure it is healthy before any change: pvecm status Look for a quorum and no nodes marked as offline. Expected output includes: Quorum information ------------------ Date: Mon Jan 1 12:00:00 2024 Quorum provider: corosync_votequorum Nodes: 3 Node ID: 0x00000001 Ring ID: 1.abc Quorate: Yes If Quorate is No , stop and fix the cluster before proceeding. ### List Running VMs and Containers You need to know what is running on the node you plan to touch. qm list For containers: pct list Both commands show IDs, names, status, and resource allocation. Record this list; you will use it to verify that everything comes back after the operation. ### Check Storage Health Proxmox uses storage plugins like LVM, ZFS, Ceph, or NFS. Check the status of your storage backends: pvesm status Expected output shows each storage with its type, status, and usage. If any storage is not active , investigate before proceeding. For ZFS pools: zpool status For Ceph: ceph -s ### Identify the Exact Component Being Inspected Do not upgrade everything at once. Pick one node at a time, or one component, and keep the change scope minimal. For example, if you are upgrading Proxmox VE from 7.x to 8.x, do not simultaneously change storage configuration or network settings. ## Safe Configuration Path A safe configuration path builds on the inventory. It means making the smallest justified change, verifying it, and having a rollback plan. ### Repository Configuration Proxmox upgrades rely on apt repositories. Ensure your sources are correct. For Proxmox VE 8.x, the enterprise repository is disabled by default if you do not have a subscription. The no-subscription repository is commonly used for testing and homelabs. Check your current repository configuration: grep -r "" /etc/apt/sources.list /etc/apt/sources.list.d/ For a proper Proxmox VE 8.x setup without a subscription, you should have a file like /etc/apt/sources.list.d/pve-no-subscription.list containing: deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription If you are using the enterprise repository, it should point to: deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise Important: mixing enterprise and no-subscription repositories can cause version conflicts. The safe path is to use one or the other. If you have an active subscription, use enterprise; otherwise, use no-subscription. ### Pre-Upgrade Health Checks Before upgrading, run a full package update and upgrade: apt update apt list --upgradable Review the list. Pay attention to any packages that might break compatibility, such as kernel, QEMU, or Ceph components. Also, check disk space on the root filesystem: df -h / Ensure you have at least 5 GB free. An upgrade that runs out of space can leave the system in an inconsistent state. ### Backup Critical Configuration Back up your configuration files before modifying them. For example, back up the Proxmox VE configuration directory: tar czf /root/pve-config-backup-$(date +%Y%m%d).tar.gz /etc/pve /etc/apt/sources.list /etc/apt/sources.list.d This archive includes cluster configuration, storage definitions, and repository settings. Store it somewhere safe, preferably off-node. ### Perform the Upgrade To upgrade Proxmox VE itself, use: apt dist-upgrade During the upgrade, watch for prompts about configuration files. If a new version of a config file is available, Proxmox usually recommends keeping the existing one. Choose N or keep the local version unless you have a specific reason. After the upgrade, check if a reboot is required. Proxmox may install a new kernel. You can check the current running kernel and the installed kernel packages: uname -r dpkg --list | grep pve-kernel If the running kernel is older than the newest installed pve-kernel, plan a reboot during a maintenance window. ## Verification and Diagnostics After any change, you must verify that the system is in the expected state. Verification should be precise and automated where possible. ### Post-Upgrade Verification After a Proxmox upgrade, run: pveversion Confirm the new version matches your target. For example, upgrading from 7.4 to 8.1 should show pve-manager/8.1.x . Check the cluster status again: pvecm status Ensure all nodes are online and quorum is intact. Verify that all VMs and containers are running: qm list pct list Compare the output to your pre-upgrade inventory. If any VM is stopped, investigate why. Check system logs for errors during boot and service startup: journalctl -b -p err Look for messages related to storage, networking, or Proxmox services like pvedaemon , pveproxy , or pvestatd . ### Validate Storage and Backup If you use ZFS, check pool status: zpool status -v This shows any degraded or faulted devices. For Ceph, run: ceph -s Confirm that the cluster is healthy ( HEALTH_OK ). Test that backups still work. You can create a manual backup of a small test VM: vzdump 100 --mode snapshot --compress zstd --storage local Replace 100 with a non-critical VM ID. This proves that the backup subsystem and storage are functional. ### Diagnostic Commands for Common Issues If something is not right, here are some diagnostic commands: - Check Proxmox service status: systemctl status pvedaemon pveproxy pvestatd - Examine the Proxmox task log: cat /var/log/pve/tasks/active - View kernel messages: dmesg | tail -50 - Check network configuration: ip addr show cat /etc/network/interfaces These commands help isolate whether a failure is due to a service, kernel, or configuration. ## Migration Planning and Execution Migration in Proxmox can mean moving a VM or container from one node to another within a cluster, or moving a standalone node's data to a new hardware. This section covers live migration within a cluster and offline migration for standalone setups. ### Live Migration Within a Cluster Live migration moves a running VM with minimal downtime. It requires shared storage or local storage with replication. For this example, we assume a two-node cluster with ZFS replication. First, ensure the VM is replicating: pve-zsync list Or check replication jobs in the GUI. The VM must have a replication job configured. To migrate VM 100 from node A to node B: qm migrate 100 nodeB --online Watch the migration progress: qm status 100 --verbose After migration, the VM should show as running on node B. Verify by running qm list on node B. Migration may fail if there is not enough memory or if CPU types differ significantly. To mitigate, set the VM's CPU type to kvm64 before migration for maximum compatibility. ### Offline Migration for Standalone Nodes If you are moving from one standalone Proxmox node to another without shared storage, you need to back up and restore. - Create a backup of the VM: vzdump 100 --mode stop --compress zstd --storage /mnt/backup - Copy the backup file to the new node using scp or a USB drive. - On the new node, restore the backup: qmrestore /mnt/backup/vzdump-qemu-100-2024_01_01-12_00_00.vma.zst 100 - Start the VM and verify its network and services. This method has downtime equal to the backup and restore time. For critical VMs, plan accordingly. ### Storage Migration Proxmox also allows moving a VM's disk image from one storage to another while the VM is running (if the storage supports it). For example, to move VM 100's disk from local-lvm to ceph-pool : qm move-disk 100 scsi0 ceph-pool --delete The --delete flag removes the original disk after successful copy. Without it, the original is kept, which is a good safety measure. Monitor disk migration progress with: qm status 100 --verbose Look for the disk section to see the storage and size. ## Failure Modes and Recovery Despite best practices, failures happen. This section describes common failure modes and recovery steps. ### Upgrade Failure Due to Package Conflict Symptom: apt dist-upgrade fails with dependency errors or dpkg returns an error code. Recovery: - Run apt -f install to fix broken dependencies. - If that fails, check /var/log/apt/term.log and /var/log/dpkg.log for the specific package. - You may need to remove or hold a problematic package. For example: apt remove problematic-package - After fixing, rerun apt dist-upgrade . ### Kernel Panic After Reboot Symptom: After upgrading and rebooting, the node does not boot, showing a kernel panic or hanging. Recovery: - Reboot and select the previous kernel from the GRUB menu (Advanced options for Proxmox VE). - Once booted with the old kernel, pin the good kernel and remove the problematic one: proxmox-boot-tool kernel pin 5.15.107-2-pve - You can also remove the bad kernel with apt remove pve-kernel- . - Investigate the cause before attempting the upgrade again. ### Migration Failure Due to Network or Storage Symptom: qm migrate fails with a timeout or I/O error. Recovery: - Check network connectivity between nodes: ping -c 4 nodeB - Verify that the migration network is configured and not blocked by firewall. - Check storage availability on both nodes. For ZFS replication, ensure the replication target has enough space and the pool is healthy. - If the VM is still running on the source node, you can retry after fixing the issue. If the VM is in an undefined state, stop it on the source node and start it on the target manually after ensuring the disk data is consistent. ### Corrupted Configuration Symptom: Proxmox services fail to start due to configuration errors. Recovery: - Restore from the configuration backup you made: tar xzf /root/pve-config-backup-.tar.gz -C / - Restart Proxmox services: systemctl restart pvedaemon pveproxy pvestatd - Verify with pveversion and pvecm status . ## Operations Checklist Use this checklist before, during, and after any Proxmox upgrade or migration. Each item has an owner and a review frequency.
StepTaskCommand / CheckExpected ResultOwnerReview When
1Record current versionpveversionVersion string matches expectedPriya Shah, Engineering LeadBefore any change
2Check cluster quorumpvecm statusQuorate: YesPriya ShahBefore change and after
3List VMs and CTsqm list; pct listAll intended VMs runningAlex Chen, DevOps EngineerBefore and after
4Validate storage healthpvesm status, zpool status, ceph -sAll storage active, no degraded poolsMaria Garcia, Storage AdminBefore and after
5Backup configurationtar czf ...Backup file created and stored off-nodeAlex ChenBefore upgrade
6Check repository configgrep -r "" /etc/apt/sources.list /etc/apt/sources.list.d/Only intended repos enabledPriya ShahBefore upgrade
7Run package upgradeapt dist-upgradeNo errors, packages upgradedAlex ChenDuring maintenance window
8Reboot if kernel updateduname -r vs dpkg --list pve-kernel*Running kernel is latestAlex ChenAfter upgrade
9Verify services and VMssystemctl status pvedaemon, qm listAll services active, VMs runningPriya ShahAfter upgrade/migration
10Test backup and restorevzdump <test-vm>Backup completes, restore testedMaria GarciaAfter upgrade or before high-risk change
11Validate cluster and storage post-changepvecm status, ceph -sNo errors, healthy statusMaria GarciaAfter any operation
12Document outcome and lessons learnedUpdate runbookRunbook currentPriya ShahAfter each operation
Assign a single accountable owner for each checklist item. The owner is responsible for executing and verifying the step, and for escalating if something goes wrong. Review the entire checklist during a weekly operations meeting. ## Common Pitfalls and How to Avoid Them ### 1. Skipping the Inventory Phase Why it happens: Teams are under time pressure and want to "just upgrade." They skip checking versions, storage health, or cluster quorum. How to avoid: Make the inventory a mandatory gate. Use the checklist above and require sign-off from the accountable owner before proceeding. Recovery if hit: Stop immediately, document current state as best you can, and restore from backups if necessary. ### 2. Using Mixed Repositories Why it happens: Someone enables both enterprise and no-subscription repositories, or adds a Debian repo without understanding dependency conflicts. How to avoid: Use only one Proxmox repository per major version. Remove or comment out others. Test in a staging environment if possible. Recovery: Disable the conflicting repo, run apt update , then apt -f install , and consider downgrading specific packages if needed. ### 3. Insufficient Disk Space Why it happens: The root filesystem fills up during a large upgrade, causing dpkg to fail mid-installation. How to avoid: Before upgrading, check df -h / and clean up old kernels, logs, and unused packages. Keep at least 5 GB free, more if you have many packages. Recovery: Boot into recovery mode or use a live CD to free space. Then run apt -f install to complete the upgrade. If the system is unbootable, restore from backup. ### 4. Not Testing Backups Before Migration Why it happens: Teams assume backups work because they are scheduled, but never actually restore. How to avoid: Perform a test restore of a non-critical VM before the migration window. Automate this test periodically. Recovery: If you discover the backup is corrupt during migration and the source is gone, you may lose data. To reduce risk, take a fresh backup immediately before migration and verify its integrity with vzdump --verify . ### 5. Ignoring Replication Health for Live Migration Why it happens: The replication job has been failing silently, so the target node does not have an up-to-date copy of the VM disk. How to avoid: Check replication status before migrating. Run pve-zync list or check the Proxmox GUI for replication logs. Ensure last sync is recent. Recovery: Manually trigger a replication sync or use offline migration if replication is not reliable. ### 6. Not Having a Rollback Plan Why it happens: Overconfidence or lack of experience leads to no fallback. How to avoid: Always define a rollback path before making a change. For upgrades, know how to boot the previous kernel. For migration, know how to fail back to the original node. Document these steps in a runbook. Recovery: If you are stuck without a plan, minimize damage by isolating the affected node, restoring from the latest backup, and seeking expert help. ## Conclusion Proxmox upgrade and migration are manageable when approached with discipline: inventory first, change minimally, verify rigorously, and always have a recovery path. The commands and checklists in this guide give you a concrete operational framework. As a next step, take one low-risk action: run pveversion and pvecm status on each of your nodes and record the output. Then review your backup and restore procedure with a test restore. This simple practice will reveal gaps before you attempt a major upgrade or migration. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Use this guide as your operational baseline, and adapt the specifics to your environment.