Proxmox monitoring and alerts with practical examples: practical implementation guide

Intro

Proxmox powers critical virtual machines, storage, and networks. When it falters, customers feel it immediately. This guide shows exactly what to monitor, how to build useful dashboards, which alert rules to start with, and how to respond when things go wrong. It is written for developers, DevOps consultants, and startup teams who need reliable operations without a large platform team.

The approach is hands-on: concrete metrics, copy-pasteable alert rules, and short runbooks you can test locally.

Workflow Overview

A clear monitoring workflow keeps noise low and action high:

1. Define goals and priorities

• State your top risks: data loss, downtime, noisy neighbors, failed backups.
• Set simple targets: e.g., no more than 0.1% packet loss, backups succeed daily, Ceph is healthy.

1. Instrument and collect

• Metrics: enable standard Linux node metrics and export Proxmox service states. If you use Ceph, export a 1 or 0 for health.
• Logs: forward journald/syslog. Parse vzdump and Ceph health lines.

1. Alert design

• Start with a small set of high-signal alerts. Tie each to a runbook and owner.
• Use multi-window thresholds and for: durations to avoid flapping.

1. Dashboards

• Build fast triage views for node, storage, VM, and cluster health.

1. Drill and iterate

• Simulate CPU pressure, disk fill, and failed backups in a lab. Tune until alerts are timely and actionable.

A staged plan reduces rework and speeds up adoption.

What to Monitor

Cover the layers that Proxmox coordinates. Prioritize signals that are predictive and actionable.

• Node health (Linux)
• CPU: saturation and steal time (noisy neighbor signal)
• Memory: available, swap activity, page faults
• Load average vs CPU count (sustained > cores often degrades VMs)
• Uptime and reboot reasons

• Storage
• Filesystem fullness and inode exhaustion on VM images and volumes
• IO latency and queue depth, per device (exclude loop, ram, temp devices)
• ZFS pool status and resilvers, LVM-thin data pool usage
• SMART warnings and reallocations
• Ceph: health, MON quorum, OSD up/in counts, PG states, recovery, latency

• Networking
• Interface errors and drops on bridges and bonds
• Packet loss and MTU mismatches on uplinks
• VRRP or gateway reachability for HA northbound paths

• Proxmox services and cluster
• corosync and pve-ha states, cluster quorum
• pveproxy/pvedaemon availability
• Live migration success rate and durations

• Workloads (VMs and containers)
• VM CPU steal, CPU ready-like symptoms under contention
• Disk and network throughput per VM
• QEMU guest agent status for better visibility

• Backup and restore
• vzdump success rate, duration, and size trends
• Restore tests on a schedule (smoke test at least weekly)

• Security and change signals
• Login failures, sudo anomalies, unexpected reboots
• Package updates pending on hypervisors

Useful Metrics and Queries

Below are actionable metrics and example queries that work well on Proxmox hosts running standard Linux exporters. Adjust label filters for your environment.

• CPU saturation per node

# Percent busy CPU over 5 minutes
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

• CPU steal (oversubscription or noisy neighbor)

avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m]) * 100)

• Memory pressure

(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.10

• Swap activity

rate(node_vmstat_pswpin[5m]) + rate(node_vmstat_pswpout[5m])

• Filesystem saturation (ext4/xfs/zfs)

(node_filesystem_avail_bytes{fstype=~"ext4|xfs|zfs",mountpoint!~"/run|/var/lib/docker|/snap"}
 /
 node_filesystem_size_bytes{fstype=~"ext4|xfs|zfs",mountpoint!~"/run|/var/lib/docker|/snap"}) < 0.15

• Disk latency approximation (average read and write)

# Avg read latency seconds per op
(rate(node_disk_read_time_seconds_total{device!~"^(ram|loop|fd|sr|dm).*"}[5m])
 /
 clamp_min(rate(node_disk_reads_completed_total{device!~"^(ram|loop|fd|sr|dm).*"}[5m]), 1))

# Avg write latency seconds per op
(rate(node_disk_write_time_seconds_total{device!~"^(ram|loop|fd|sr|dm).*"}[5m])
 /
 clamp_min(rate(node_disk_writes_completed_total{device!~"^(ram|loop|fd|sr|dm).*"}[5m]), 1))

• Network errors

(rate(node_network_receive_errs_total{device!~"^(lo|veth).*"}[5m]) > 0)
 or
(rate(node_network_transmit_errs_total{device!~"^(lo|veth).*"}[5m]) > 0)

• Systemd service health (corosync example)

node_systemd_unit_state{name="corosync.service",state="active"} != 1

Export daily counters from vzdump logs into metrics:

• Backup results via textfile collector

# /usr/local/bin/vzdump_metrics.sh
#!/bin/sh
ok=$(grep -c "Finished Backup" /var/log/vzdump/* 2>/dev/null || true)
err=$(grep -c "ERROR" /var/log/vzdump/* 2>/dev/null || true)
echo "pve_vzdump_jobs_ok $ok"
echo "pve_vzdump_jobs_error $err"

Run via cron and drop output to node_exporter textfile directory:

* * * * * root /usr/local/bin/vzdump_metrics.sh > /var/lib/node_exporter/textfile_collector/pve.prom

Useful query:

increase(pve_vzdump_jobs_error[24h]) > 0

• Ceph health as a simple metric

# /usr/local/bin/ceph_health_metric.sh
#!/bin/sh
h=$(ceph health | awk '{print $1}')
if [ "$h" = "HEALTH_OK" ]; then echo "ceph_health 1"; else echo "ceph_health 0"; fi

Metric rule:

ceph_health == 0

Logs and Signals

Centralize journald/syslog from all Proxmox nodes and parse high-value patterns.

Key log sources and patterns:

• Cluster and HA
• corosync: lost quorum, token timeouts
• pve-ha-crm: fenced, service failover, recovery complete
• Storage
• kernel: I/O error, EXT4-fs error, ZFS pool degraded, rpool resilver
• smartd: Pre-fail, reallocated sector
• Ceph: mon down, osd out, degraded, slow requests
• VM lifecycle
• qm: migration failed, task error
• Backups
• vzdump: ERROR, exit code, tar or zstd failures
• Security
• sshd: Failed password, invalid user

Useful ad-hoc filters during triage:

journalctl -u corosync --since "-1h"
journalctl -t pve-ha-crm --since "-1h" | egrep -i 'fail|fence|quorum'
journalctl -k --since "-1h" | egrep -i 'error|i/o|ext4|zfs|nvme|sda'
grep -i 'ERROR' /var/log/vzdump/* | tail -n 50

Dashboards

Design dashboards that answer the on-call engineer's first three questions: what broke, where, and how bad.

• Node overview (one row per node)
• CPU busy and steal, memory available, swap activity
• Filesystem usage for VM images and backup targets
• Top disk latency devices, network errors by interface
• corosync.service state and pve-ha signals

• Cluster health
• Quorum state, node availability
• Live migrations count and median duration
• Backup success vs failure over time

• Storage detail
• ZFS pool capacity and resilver activity, slow disks
• Ceph: health flag, OSD up/in, PG states, read/write latency

• VM and tenant view
• Per-VM CPU, memory, IO, and network
• Busy neighbors list to spot contention

Design tips:

• Keep red metrics at the top left. Put derivatives and rates next to the raw values.
• Add links from panels to runbooks and to host consoles.
• Use consistent colors and units across dashboards.

Alert Rules

Start with a compact, actionable set. Example Prometheus-style rules below.

• Node CPU saturation

- alert: NodeHighCPU
  expr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: High CPU on {{ $labels.instance }}
    description: CPU > 85% for 10m. Check top VMs and processes.

• CPU steal (contention)

- alert: NodeHighCPU_Steal
  expr: avg by (instance) (rate(node_cpu_seconds_total{mode="steal"}[5m]) * 100) > 5
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: CPU steal on {{ $labels.instance }}
    description: Host CPU contention. Investigate noisy VMs and scheduling.

• Low memory available

- alert: NodeLowMemory
  expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.10
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: Low memory on {{ $labels.instance }}
    description: Less than 10% memory available.

• Filesystem nearly full (exclude ephemeral mounts)

- alert: FilesystemNearlyFull
  expr: (node_filesystem_avail_bytes{fstype=~"ext4|xfs|zfs",mountpoint!~"/run|/var/lib/docker|/snap"}
        / node_filesystem_size_bytes{fstype=~"ext4|xfs|zfs",mountpoint!~"/run|/var/lib/docker|/snap"}) < 0.15
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: Filesystem low free space on {{ $labels.instance }} {{ $labels.mountpoint }}
    description: Less than 15% free. Clean up or expand capacity.

• Disk latency high

- alert: DiskLatencyHigh
  expr: (rate(node_disk_read_time_seconds_total{device!~"^(ram|loop|fd|sr|dm).*"}[5m])
        / clamp_min(rate(node_disk_reads_completed_total{device!~"^(ram|loop|fd|sr|dm).*"}[5m]),1)) > 0.02
        or
        (rate(node_disk_write_time_seconds_total{device!~"^(ram|loop|fd|sr|dm).*"}[5m])
        / clamp_min(rate(node_disk_writes_completed_total{device!~"^(ram|loop|fd|sr|dm).*"}[5m]),1)) > 0.02
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: High disk latency on {{ $labels.instance }} {{ $labels.device }}
    description: Avg latency > 20ms for 5m. Check device health and workload.

• Network errors present

- alert: NetworkErrors
  expr: rate(node_network_receive_errs_total{device!~"^(lo|veth).*"}[5m]) > 0 or rate(node_network_transmit_errs_total{device!~"^(lo|veth).*"}[5m]) > 0
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: Network errors on {{ $labels.instance }} {{ $labels.device }}
    description: Interface is reporting errors. Check cabling, drivers, and MTU.

• Corosync not active

- alert: CorosyncDown
  expr: node_systemd_unit_state{name="corosync.service",state="active"} != 1
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: Corosync inactive on {{ $labels.instance }}
    description: Cluster communication down. Risk of losing quorum.

• Backup failures detected (from textfile counter)

- alert: BackupFailures
  expr: increase(pve_vzdump_jobs_error[24h]) > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: Backup failures on {{ $labels.instance }}
    description: One or more backups failed in the last 24h. Investigate logs.

• Ceph health not OK (from simple metric)

- alert: CephUnhealthy
  expr: ceph_health == 0
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: Ceph health is not OK
    description: Check MON quorum, OSD states, and PGs.

Severity mapping suggestion:

• page: CorosyncDown, FilesystemNearlyFull on VM storage, CephUnhealthy
• ticket: NodeHighCPU, NodeLowMemory, DiskLatencyHigh, NetworkErrors
• notification: BackupFailures (if daily) escalates to page if 2 consecutive days

Incident Response

Keep runbooks short and tied to alert names.

• FilesystemNearlyFull

1. Identify top talkers: du -xhd1 /var/lib/vz; zfs list -o name, used, avail
2. Prune old backups or move them to external storage
3. If LVM-thin is full, add space or migrate VMs
4. For ZFS, consider zfs set compression=lz4 on bulky datasets

• CorosyncDown

1. Check network links on the node (bond/bridge) and switch status
2. Validate time sync and multicast or UDP reachability
3. Review journalctl -u corosync; restart only if safe and peers are healthy
4. If multiple nodes impacted, prioritize restoring quorum

• CephUnhealthy

1. ceph -s to identify PG states and missing OSDs
2. Bring OSDs back in; check disks and power
3. Defer nonessential rebalances during peak hours

• NodeHighCPU and steal

1. Identify chatty VMs: qm list, per-VM metrics
2. Migrate or throttle offenders; adjust CPU shares
3. Revisit consolidation ratios

• BackupFailures

1. Inspect /var/log/vzdump and target storage capacity
2. Retry a small VM backup to isolate network vs storage
3. Validate credentials and retention policies

General flow for any alert:

• Confirm scope, stabilize the platform, restore customer impact, then repair root cause.
• Annotate dashboards with remediation taken and follow-ups.

Local Pilot Plan

A small, verifiable pilot catches most issues early and keeps risk low.

Scope

• 1 Proxmox node, 1 busy VM, 1 storage pool (ZFS or LVM), and existing backups

Metrics and logs

• Enable Linux node metrics
• Add the vzdump and ceph textfile scripts if applicable
• Forward journald/syslog

Alerts to enable on day 1

• NodeHighCPU, NodeLowMemory, FilesystemNearlyFull, NetworkErrors, BackupFailures

Dashboard

• Single-page node view with CPU, memory, filesystem, IO latency, network errors, and service states

Verification timeline (1 day)

• CPU: stress-ng or a compile to push CPU > 85% for 10m
• Memory: start a big tmpfs to cross the 10% available threshold
• Filesystem: create a large file on the VM storage volume until 85% usage, then remove
• Network: briefly flap a lab link or use tc to inject errors
• Backup: run a manual vzdump; confirm metrics and logs update

Exit criteria

• All 5 alerts fire under test, notify the right channel, and resolve when conditions clear
• Dashboard shows correlated spikes with readable units and legends
• Runbooks are updated with what worked

This narrow, measurable pilot is easy to inspect locally before rolling out to the cluster.

Conclusion

You now have a practical plan to monitor Proxmox: what to watch, ready-to-use alert rules, log patterns that matter, and dashboards that speed up triage. Start with the local pilot, validate end to end, then broaden coverage to Ceph, more nodes, and more VMs. Keep alerts few and actionable, connect them to short runbooks, and iterate based on real incidents.