Proxmox Production Operations Checklist: A Practical Implementation Guide

Introduction

Operating Proxmox Virtual Environment (Proxmox VE) in production requires more than knowing how to create a virtual machine. It demands a disciplined approach to observing the current state, making minimal changes, and verifying outcomes. This article provides a practical operations checklist for Proxmox production environments, aimed at developers, DevOps consultants, and technical startup teams. Each section includes concrete commands, expected outputs, failure signals, and recovery steps, all grounded in real-world scenarios.

The core principle is operational safety: observe before changing, limit the blast radius of any intervention, use placeholders instead of real credentials, verify the result, and document recovery paths before you need them.

Version and Environment Inventory

Before making any change, you need a complete picture of your Proxmox environment. This includes the installed version, cluster topology, storage configuration, and hardware or virtual resources. A read-only command from Proxmox's CLI is the safest way to capture this information.

Start by identifying the Proxmox VE version:

pveversion

Expected output example:

pve-manager/7.4-3/9002ab8a (running kernel: 5.15.107-2-pve)

This tells you the exact version and kernel, which is critical for compatibility checks and documentation.

Next, check the cluster status (if applicable):

pvecm status

Expected output for a healthy single node:

Cluster information
-------------------
Name:             mycluster
Config Version:   4
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Mon Apr 15 10:30:00 2024
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1.1a
Quorate:          Yes

For a multi-node cluster, you should see each node listed with its status and the quorum should be 'Yes'. If quorum is 'No', the cluster is not healthy, and writes to shared storage may be blocked.

Check resource usage of nodes and guests:

pvesh get /cluster/resources --output-format yaml

This API call returns a structured list of all nodes, VMs, and storage with their current CPU, memory, and disk usage. Parse it with a script or inspect manually for anomalies.

Finally, list all configured storage:

pvesm status

Expected output includes each storage's name, type, status, and usage. Any storage marked 'inactive' or with high usage (over 85%) needs attention.

Practical Example: Morning Health Check

Every morning, a designated operations engineer (e.g., Priya Shah, Infrastructure Lead) runs a script that executes these commands and compares outputs against expected baselines. The engineer is responsible for investigating any deviation. This check is revisited in the daily standup and weekly ops review.

Safe Configuration Path

Changing Proxmox configuration can affect many VMs at once, so safety is paramount. The process should be: capture current state, back up relevant files, make one scoped change, verify, and know how to roll back.

Example 1: Changing VM Memory

Suppose you need to increase the memory of VM 100 from 4 GB to 8 GB. The VM is running a critical web application.

First, observe current configuration:

qm config 100 | grep memory

Expected:

memory: 4096

Check current RAM usage of the VM (from inside the guest or via Proxmox monitoring). Ensure the host has enough free memory. Then, make the change:

qm set 100 -memory 8192

Verify immediately:

qm config 100 | grep memory

Expected:

memory: 8192

Note: For a running VM, the change may not take effect until reboot unless ballooning is enabled. Check the VM's balloon driver and reboot if necessary. Rollback: set memory back to 4096 with the same command.

Example 2: Adding a New User

Creating a new user in Proxmox with specific permissions:

pveum useradd devops@pve --password 'SecurePlaceholder123!' --comment 'DevOps engineer'
pveum aclmod /vms/100 -user devops@pve -role PVEVMUser

Verify:

pveum user list
pveum acl list

Rollback: remove the user and ACL with pveum userdel devops@pve.

Important Precautions

• Always back up configuration files before editing them directly. For example, before editing /etc/pve/storage.cfg, copy it: cp /etc/pve/storage.cfg /etc/pve/storage.cfg.bak-$(date +%Y%m%d).
• Use the official CLI or API rather than editing files manually, when possible.
• Test changes on a non-production node or in a test cluster first.
• Document the change in your change management system.

Verification and Diagnostics

Verification is not a one-time step; it should be continuous and integrated into daily operations. Here are key diagnostics for Proxmox production.

System Logs

Proxmox logs are essential for troubleshooting. The main log file is /var/log/syslog. You can also use journalctl:

journalctl -u pve-cluster -u pvedaemon -u pveproxy -u pvestatd --since "1 hour ago"

Look for errors like 'connection timed out', 'storage is not online', or 'quorum lost'.

Network Diagnostics

Check the status of network bridges and interfaces:

ip addr show vmbr0

Expected: interface is up with an IP address. If not, VMs on that bridge may lose connectivity.

Test connectivity to a critical storage or backup server:

ping -c 4 storage.example.com

VM Status and Performance

List all VMs and their states:

qm list

Watch live resource usage:

qm monitor 100

Inside the qm monitor, type info block to see disk stats, info cpus for CPU info, and help for more commands.

Cluster Health (if applicable)

For clusters, check corosync and quorum:

corosync-quorumtool -s

Expected: 'Quorate: Yes' and all nodes listed.

Backup Verification

For each backup job, verify the backup completed successfully:

cat /var/log/pve/tasks/active

Or check the backup status in the GUI. Test restores periodically to ensure backups are valid.

Failure Modes and Recovery

Understanding common failure modes and having ready recovery procedures is critical.

Failure: Loss of Quorum in Cluster

Cause: Network partition or node failure.

Symptoms: Cluster operations fail, VMs may not start or migrate, web GUI may show errors.

Recovery:

1. Identify which nodes can still see each other.
2. Console into each node and check corosync status: systemctl status corosync.
3. If a node is physically down, wait for it to recover, or remove it from the cluster if it's permanently lost: pvecm delnode <nodename> (only if you have quorum on remaining nodes).
4. Regain quorum: if using two-node cluster, ensure the qdevice is functioning. Check corosync-quorumtool.
5. After restoring quorum, verify cluster consistency: pvecm status.

Failure: Storage Not Available

Cause: NFS server down, iSCSI target unreachable, or local disk failure.

Symptoms: VMs show I/O errors, storage marked 'inactive' in pvesm status.

Recovery:

1. Check network connectivity to storage server: ping <storage-IP>.
2. Check storage status: pvesm status.
3. For NFS, check mounts: mount | grep nfs.
4. If NFS server is back, Proxmox should reconnect; otherwise, manually mount: mount -a.
5. For local disk, check SMART data: smartctl -a /dev/sda (if installed).
6. If disk is failing, migrate VMs to other storage or nodes.

Failure: Backup Fails

Cause: Insufficient space, network issue, or snapshot problem.

Recovery:

1. Check backup log: /var/log/pve/tasks/[UPID].log.
2. Check destination storage space: df -h on backup server.
3. If using vzdump with snapshot mode and snapshot fails, try suspend mode or stop mode.
4. Test with a manual backup: vzdump 100 --mode snapshot --storage backupstore.
5. Review retention settings to free space.

Operations Checklist

Here is a consolidated checklist for daily, weekly, and monthly operations, with accountable owners and review frequency. Adapt to your environment.

Daily Checks (Owner: On-call engineer, reviewed in daily standup)

• [ ] Run pveversion and compare against expected version.
• [ ] Check cluster status with pvecm status (if clustered).
• [ ] Run pvesh get /cluster/resources and review for high usage (>85% on any resource).
• [ ] Check storage status with pvesm status.
• [ ] Review system logs for errors: journalctl -p err -b.
• [ ] Verify all critical VMs are running (qm list).
• [ ] Check backup jobs completed successfully.

Weekly Checks (Owner: Infrastructure Lead, reviewed in weekly ops meeting)

• [ ] Review performance trends (CPU, memory, disk I/O) for nodes and top VMs.
• [ ] Check disk space growth and plan for expansion.
• [ ] Apply security patches after testing (see Safe Configuration Path).
• [ ] Validate backup integrity by restoring a test VM from the latest backup.
• [ ] Review user accounts and permissions (pveum user list, pveum acl list).
• [ ] Check for hardware warnings (IPMI, SMART).

Monthly Checks (Owner: Infrastructure Lead, reviewed in monthly architecture review)

• [ ] Perform full capacity planning: project future needs based on trends.
• [ ] Test disaster recovery plan: simulate node failure and recover.
• [ ] Review Proxmox subscription and support.
• [ ] Document any configuration changes and update runbooks.
• [ ] Test high availability failover if configured: ha-manager status and ha-manager migrate <vm> <node>.

Common Pitfalls and How to Avoid Them

1. Not Documenting Baselines

Why it happens: Teams skip recording normal state, making it impossible to detect anomalies.

How to avoid: Use the daily checklist to capture key metrics into a monitoring system (e.g., Prometheus, Zabbix). Set up alerts for deviations.

2. Making Changes Without Testing on a Non-Production Environment

Why it happens: Pressure to fix issues quickly.

How to avoid: Maintain a test cluster or at least a test node. Document test results before applying changes to production.

3. Ignoring Backup Verification

Why it happens: Backups are configured, but restores are never tested.

How to avoid: Schedule monthly restore tests. Automate if possible. Remember: a backup is only valid if you can restore from it.

4. Using Default or Weak Credentials

Why it happens: Convenience.

How to avoid: Enforce strong passwords, two-factor authentication for web access, and use API tokens with limited permissions.

5. Overlooking Storage Health

Why it happens: Focus on VMs, forgetting underlying storage.

How to avoid: Monitor storage SMART attributes and filesystem usage. Set alerts for high usage or degraded RAID.

6. Uncontrolled Changes

Why it happens: Multiple admins making changes without coordination.

How to avoid: Implement a change management process with approvals. Use Proxmox's task log to track changes (/var/log/pve/tasks/index).

Conclusion

Running Proxmox VE in production requires a structured approach to operations. By using version-specific, observable, and reversible procedures, you can minimize downtime and ensure reliability. The checklist provided here is a starting point; adapt it to your environment's specific needs.

Start with one low-risk verification: run the morning health check tomorrow. Record the current state, compare it with expected signals, and review any anomalies. From there, build up to the full checklist and integrate it into your team's routine.

Remember: a reliable workflow makes failures visible, protects sensitive information, limits changes to the intended scope, and defines recovery verification before an incident occurs.

---

Note: All commands and outputs are examples for Proxmox VE 7.x. Always refer to the official documentation for your specific version.