Proxmox CI/CD Automation: A Practical Field Guide

Intro

Automating Proxmox with CI/CD means treating virtual machines like code: every VM is defined in a versioned repository, every change goes through review and automated checks, and every deployment leaves an audit trail. This guide is for developers, DevOps engineers, and technical startup teams who want to replace manual Proxmox clicks with repeatable pipelines.

You will learn how to:

• Inventory your Proxmox environment and plan automation safely.
• Build a minimal but production-ready CI/CD pipeline with GitLab CI, Terraform, and Ansible.
• Deploy and configure a VM from zero to serving traffic.
• Add health checks, rollback plans, and failure diagnosis.
• Avoid the mistakes that make Proxmox automation brittle and dangerous.

Every example uses explicit placeholders (<placeholder>) and assumes Proxmox VE 8.x. Do not copy-paste real credentials, tokens, or production identifiers into version control.

The goal is operational safety: observe before changing, limit the blast radius, verify each step, and know how to recover before you need to.

Version and Environment Inventory

Before writing a single line of pipeline code, collect the facts about your Proxmox installation. This prevents version mismatches and wrong API assumptions.

What to capture

Run these read-only commands on a Proxmox node via SSH or the web shell:

# Proxmox version
pveversion
# Expected output: pve-manager/8.1.4/7d9c1b5e (running kernel: 6.5.11-7-pve)

# Cluster status (if clustered)
pvecm status
# Expected output: Quorum information, member list, and each node marked as online

# Storage available for VM disks
pvesm status
# Expected output: A table of storage IDs, types, and usage. Note the storage type: dir, lvm, lvmthin, zfs, cephfs, nfs, etc.

# Network bridges
ip link show | grep -E '^[0-9]+: (vmbr|tap)'
# Expected output: Lines showing vmbr0, vmbr1, etc., with state UP

Record these in a environment.md file in your automation repository:

# Proxmox Environment Snapshot
Date: 2025-02-20
Node: pve-node-1
PVE version: 8.1.4
Cluster: yes (3 nodes, quorum OK)
Storage for VMs: local-lvm (lvmthin), ceph-pool (cephfs)
Network bridges: vmbr0 (management), vmbr1 (VM public)
Automation user: terraform@pve (API token only)

Automating the inventory

You can automate this inventory with a scheduled script. Example using Ansible:

# playbooks/facts.yml
- name: Collect Proxmox facts
  hosts: pve_nodes
  gather_facts: false
  vars:
    ansible_user: root
    ansible_python_interpreter: /usr/bin/python3
  tasks:
    - name: Run pveversion
      command: pveversion
      register: pve_version

    - name: Show version
      debug:
        msg: "{{ pve_version.stdout }}"

Run it with ansible-playbook -i inventory/proxmox.yml playbooks/facts.yml. Schedule it weekly or in your pipeline's first stage to detect drift.

Why this matters

If you write Terraform code against Proxmox API v2 and your server is still on API v1, everything fails. If you assume local-lvm is available but the node uses ZFS, your disk definitions break. Inventory first, automate second.

Safe Configuration Path

Automation credentials should follow least privilege. Instead of using the root password, create a dedicated Proxmox user with API token permissions.

Create a restricted API token

On the Proxmox web UI:

1. Go to Datacenter > Permissions > Users and click Add.

• Username: terraform@pve
• Realm: PVE Authentication
• Password: generate a strong one, store in a secrets manager.

1. Go to Datacenter > Permissions > API Tokens and click Add.

• User: terraform@pve
• Token ID: ci-token
• Uncheck Privilege Separation (so the token has the user's permissions).
• Note the full token ID: terraform@pve!ci-token and the secret (it is shown only once).

1. Go to Datacenter > Permissions > Add > API Token Permission.

• Path: /
• Token: terraform@pve!ci-token
• Role: PVEVMAdmin (allows VM management but not host-level changes).

This token can create, start, stop, and delete VMs, but cannot modify storage or network settings. It is suitable for CI/CD.

Store secrets safely

Never put the token secret in the pipeline file. In GitLab CI, use masked variables:

• PROXMOX_URL = https://pve-node-1.example.com:8006/api2/json
• PROXMOX_TOKEN_ID = terraform@pve!ci-token
• PROXMOX_TOKEN_SECRET = (the long secret string, masked)

You can also use HashiCorp Vault, AWS Secrets Manager, or GitLab CI Variables. The token should be rotated every 90 days, and the rotation should be part of your operations checklist.

Verify token access

Test the token with curl:

curl -k -H "Authorization: PVEAPIToken=terraform@pve!ci-token=<SECRET>" \
  https://pve-node-1.example.com:8006/api2/json/nodes
# Expected: JSON array of nodes

A 401 or 403 error means the token is misconfigured.

Building the CI/CD Pipeline with Terraform and GitLab CI

We will use Terraform to define a Proxmox VM, Ansible to configure it, and GitLab CI to orchestrate everything. The same pattern works with GitHub Actions, Jenkins, or any runner that can execute containers.

Repository structure

proxmox-ci/
├── .gitlab-ci.yml
├── terraform/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
├── ansible/
│   ├── site.yml
│   └── inventory/
│       └── hosts.yml
└── scripts/
    └── healthcheck.sh

Terraform provider configuration

terraform/main.tf:

terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "0.60.0"
    }
  }
}

provider "proxmox" {
  endpoint = var.proxmox_url
  insecure = true # only for self-signed certs; set false with valid CA
  api_token = "${var.proxmox_token_id}=${var.proxmox_token_secret}"
}

terraform/variables.tf:

variable "proxmox_url" {
  type    = string
  sensitive = true
}
variable "proxmox_token_id" {
  type    = string
  sensitive = true
}
variable "proxmox_token_secret" {
  type    = string
  sensitive = true
}
variable "vm_name" {
  type    = string
  default = "web-demo"
}
variable "vm_cpu" {
  type    = number
  default = 2
}
variable "vm_memory" {
  type    = number
  default = 2048
}

VM definition

Add a VM resource to main.tf using the Ubuntu cloud image template:

resource "proxmox_virtual_environment_vm" "web" {
  name      = var.vm_name
  node_name = "pve-node-1"
  vm_id     = 9001

  cpu {
    cores = var.vm_cpu
  }
  memory {
    dedicated = var.vm_memory
  }

  agent {
    enabled = true
  }

  disk {
    datastore_id = "local-lvm"
    file_id      = proxmox_virtual_environment_download_file.ubuntu_cloud_image.id
    interface    = "scsi0"
    size         = 20
  }

  network_device {
    bridge = "vmbr1"
  }

  operating_system {
    type = "l26"
  }

  initialization {
    datastore_id = "local-lvm"
    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
    user_account {
      username = "ubuntu"
      keys     = [file("~/.ssh/id_rsa.pub")]
    }
  }
}

Before using this, download a cloud image template. You can do it manually or with Terraform:

resource "proxmox_virtual_environment_download_file" "ubuntu_cloud_image" {
  content_type = "iso"
  datastore_id = "local"
  node_name    = "pve-node-1"
  url          = "https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
  file_name    = "noble-server-cloudimg-amd64.img"
}

Then convert it to a VM template. Automating template creation is beyond this article, but you can follow the Proxmox wiki for cloud-init images.

GitLab CI pipeline

.gitlab-ci.yml:

stages:
  - validate
  - plan
  - deploy
  - configure
  - verify

variables:
  TF_ROOT: terraform
  ANSIBLE_HOST_KEY_CHECKING: "false"

before_script:
  - apt-get update && apt-get install -y curl jq openssh-client
  - cd $TF_ROOT
  - terraform --version

validate:
  stage: validate
  script:
    - terraform init -backend=false
    - terraform validate
  only:
    - merge_requests

plan:
  stage: plan
  script:
    - terraform init
    - terraform plan -out=plan.tfplan -var-file=vars/prod.tfvars
  artifacts:
    paths:
      - terraform/plan.tfplan
    expire_in: 1 week
  only:
    - main

deploy:
  stage: deploy
  script:
    - terraform init
    - terraform apply -auto-approve plan.tfplan
    - terraform output -json vm_ip > /tmp/vm_ip.json
    - cat /tmp/vm_ip.json
  artifacts:
    paths:
      - /tmp/vm_ip.json
    expire_in: 1 day
  when: manual
  only:
    - main

configure:
  stage: configure
  script:
    - apt-get update && apt-get install -y ansible
    - cd ansible
    - echo "[web]" > inventory/hosts.ini
    - IP=$(jq -r '.value' /tmp/vm_ip.json)
    - echo "web ansible_host=$IP ansible_user=ubuntu ansible_ssh_private_key_file=$SSH_PRIVATE_KEY" >> inventory/hosts.ini
    - ansible-playbook -i inventory/hosts.ini site.yml -e vm_ip=$IP
  only:
    - main

verify:
  stage: verify
  script:
    - IP=$(jq -r '.value' /tmp/vm_ip.json)
    - ./scripts/healthcheck.sh $IP
  only:
    - main

Ansible configuration

ansible/site.yml:

- name: Configure web server
  hosts: web
  become: true
  tasks:
    - name: Update apt cache
      apt:
        update_cache: yes

    - name: Install nginx
      apt:
        name: nginx
        state: present

    - name: Start nginx
      service:
        name: nginx
        state: started
        enabled: yes

    - name: Create index.html
      copy:
        content: "<html><body><h1>Deployed by CI/CD</h1></body></html>"
        dest: /var/www/html/index.html
        owner: www-data
        group: www-data
        mode: '0644'

Health check script

scripts/healthcheck.sh:

#!/bin/bash
IP=$1
for i in {1..5}; do
  HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" http://$IP/)
  if [ "$HTTP_CODE" == "200" ]; then
    echo "Health check passed: HTTP 200"
    exit 0
  fi
  echo "Attempt $i: got $HTTP_CODE, retrying in 5s..."
  sleep 5
done
echo "Health check failed"
exit 1

Expected outcomes and failure signals

• terraform validate fails if syntax or provider block is invalid.
• terraform plan shows desired changes; if no changes, the VM already exists.
• deploy stage applies the plan and outputs the VM IP.
• configure stage installs nginx and starts it.
• verify stage curls the IP and expects HTTP 200.

If any stage fails, the pipeline stops. Fix the error, commit, and rerun. Terraform state is stored in GitLab's backend (you must configure a backend, e.g., HTTP or S3, or use GitLab-managed Terraform state).

Verification and Diagnostics

After deployment, always verify from outside the VM, not just from inside. The health check script already does this, but here are additional checks.

Check VM status via Proxmox API

curl -k -H "Authorization: PVEAPIToken=terraform@pve!ci-token=<SECRET>" \
  https://pve-node-1.example.com:8006/api2/json/nodes/pve-node-1/qemu/9001/status/current
# Expected: JSON with "status": "running"

Check system logs inside VM

In the Ansible playbook, add a task to retrieve logs if needed:

- name: Get nginx error log
  command: tail -n 20 /var/log/nginx/error.log
  register: nginx_log
  ignore_errors: true

- name: Show log
  debug:
    msg: "{{ nginx_log.stdout }}"

Diagnosing common issues

Always capture the exact command output before making changes. The Proxmox task log (web UI > Node > Task History) shows API actions and errors.

Failure Modes and Recovery

Automation can fail in many ways. Here are the most likely failures and how to recover.

1. Terraform state corruption

Failure: A pipeline run is interrupted while Terraform is applying. The state file on the backend may be locked or corrupted.

Recovery:

• Check if state is locked: terraform force-unlock <lock-id> (get lock ID from error).
• If state is corrupted, restore from a backup. GitLab managed state has versioning; you can rollback.
• Re-run terraform plan and then apply.

2. VM deployed but configuration fails

Failure: Terraform creates the VM, but Ansible fails halfway (e.g., apt update fails due to DNS). The VM is left in a partially configured state.

Recovery:

• Diagnose via Ansible logs. If fixable, re-run only the failed task: ansible-playbook -i inventory site.yml --start-at-task="Install nginx"
• If the VM is unusable, destroy it with Terraform: terraform destroy -target=proxmox_virtual_environment_vm.web and redeploy.
• To avoid partial state, set ignore_errors: false and use block/rescue in Ansible to clean up.

3. Rollback to previous VM version

Scenario: A new deployment introduces a broken version of the application. You need to rollback to the previous VM snapshot.

Recovery:

If you took a Proxmox snapshot before deploying, rollback is instant:

qm rollback 9001 snap_before_update
# Expected: VM is reverted to snapshot state, rebooted

You should incorporate snapshot creation into your pipeline before running configuration changes:

qm snapshot 9001 snap_before_update --description "Before app update"

If you use immutable infrastructure (treat VMs as disposable), you simply deploy the previous version tag from Git:

• On GitLab, tag each successful build with v1.2.3.
• In your pipeline, allow deploying any tag manually.
• To rollback, run the deploy job for the previous tag.

4. Accidental deletion of production VM

Failure: A Terraform change removes the VM resource because someone changed the count or name.

Recovery:

• Terraform plan would have shown the deletion. Require manual approval for apply.
• Have Proxmox backups scheduled (e.g., nightly to PBS or NFS). Restore from backup.
• Use prevent_destroy lifecycle meta-argument on critical VMs.

lifecycle {
  prevent_destroy = true
}

5. API token expires or is revoked

Failure: Pipeline fails with 401 because token was rotated or revoked.

Recovery:

• Keep token secret in a secrets manager with automatic rotation.
• In GitLab, update the CI variable and re-run the pipeline.
• Document token expiry in your runbook.

Operations Checklist

Use this checklist before and after every production deployment.

Pre-deployment checklist

Post-deployment checklist

Assign a single accountable owner for each checklist item. The owner decides when to revisit the item and signs off on completion. Use a ticketing system (Jira, GitLab issues) to track these tasks.

Common Pitfalls and How to Avoid Them

1. Using the root account for automation

Why it happens: Convenience. You need to get things done quickly. Consequence: A compromised CI token gives full control of the host. Avoidance: Create a dedicated user with API token and minimal role (PVEVMAdmin). Rotate tokens regularly.

2. Hardcoding secrets in Git

Why it happens: Developers want the pipeline to work without extra setup. Consequence: Secrets end up in commit history, exposed to anyone with repo access. Avoidance: Use CI/CD variables, Vault, or external secrets. Scan repos for secrets with git-secrets or GitLab's secret detection.

3. Not pinning provider versions

Why it happens: Using latest or omitting version in Terraform provider. Consequence: A new provider release may break your configuration. Avoidance: Pin exact versions in required_providers and use a lock file (terraform init generates .terraform.lock.hcl). Commit the lock file.

4. Skipping Terraform plan approval

Why it happens: To speed up deployments, you set auto-approve always. Consequence: A bad change can destroy production without human review. Avoidance: Use manual approval step in pipeline (when: manual) before apply. Review plan output.

5. No rollback plan

Why it happens: Teams assume new versions always work. Consequence: When a bad release goes out, panic and manual fixes cause downtime. Avoidance: Always have snapshot rollback or previous tag deploy ready. Test rollback in staging before production.

6. Ignoring cloud-init configuration

Why it happens: You create a VM from a disk image without cloud-init and expect SSH keys to work. Consequence: VM boots but you cannot log in, Ansible fails. Avoidance: Use cloud-init images and proper initialization block in Terraform. Test SSH access before proceeding.

7. Not monitoring after deployment

Why it happens: Pipeline passes, everyone moves on. Consequence: Service fails later unnoticed. Avoidance: Integrate monitoring (Prometheus, Grafana) and set up alerts. Include a verification stage that runs periodically, not just once.

Conclusion

Proxmox CI/CD automation is a powerful way to manage infrastructure as code, but it requires discipline. By following the steps in this guide—inventory, safe configuration, pipeline with Terraform and Ansible, verification, and recovery planning—you can deploy VMs reliably and roll back quickly when things go wrong.

Start with one low-risk VM, implement the pipeline, and practice a rollback. Then expand to more VMs and complex applications. Keep your automation versioned, your secrets safe, and your health checks meaningful. A reliable workflow makes failure visible and recovery a routine, not a crisis.