A Guide to HashiCorp Packer with Proxmox VE: Automating Golden Image Pipelines - ITFROMZERO - Share tobe shared! Building Proxmox VE VM templates manually causes configuration drift and stale packages. Here is our six-month production guide to automating Golden Images with HashiCorp Packer.

Table of Contents

The Real-World Issue: The Nightmare of Manual VM Provisioning at Scale

About six months ago, our team took on a project to rebuild our internal service infrastructure and scale out a Kubernetes cluster on Proxmox VE. At first, we stuck to old habits: downloading Ubuntu Server ISOs to local storage, creating empty VMs via the web GUI, and clicking through the setup wizard step by step. Next came assigning static IPs, creating users, adding SSH keys, and installing qemu-guest-agent. Once everything was set up, we converted the virtual machine into a template to clone from later.

Things started unraveling as demand grew rapidly. The team had to provision 20 to 30 new VMs every week for k3s clusters and staging environments. By that point, the template built three months prior was already missing hundreds of critical kernel security patches.

Every time an engineer cloned a new VM, they spent an extra 15 to 20 minutes running apt update && apt upgrade, installing Docker, adjusting timezones, and setting up logrotate. It peaked when a developer cloned a machine but forgot to enable and start the QEMU guest agent. Our Prometheus and Grafana monitoring went completely blind, unable to retrieve the host’s IP to collect metrics.

At home, I also run a small Proxmox homelab with 12 VMs and LXC containers to test configurations before rolling them out to production. Yet even in a homelab, taking manual snapshots and creating templates by hand is a massive chore. It needlessly ate up hundreds of gigabytes on my ZFS pool, especially when maintaining disaster recovery setups like Proxmox VE replication with ZFS. Worse still, nobody could remember which packages were installed inside each template, and the entire workflow lived completely outside Git.

Why Manual VM Templates Always Lead to Configuration Drift

After a few months of day-to-day operations, I identified three major bottlenecks with the manual point-and-click approach:

Lack of Reproducibility: Clicking through a noVNC console leaves zero audit logs. You have no way of knowing how the previous engineer configured LVM partitioning, which services were toggled on or off, or if an essential security setting was inadvertently skipped.
Configuration Drift: Golden templates become frozen in time, while upstream software packages, SSL certificates, and kernel patches constantly evolve. The longer a template sits idle, the wider the gap grows between it and the desired production state.
Skipping ISO Checksum Verification: Downloading ISOs from arbitrary mirror servers introduces the risk of corrupted or tampered files. In practice, engineers frequently skip SHA-256 validation because SSH-ing into the hypervisor to run long hashing commands feels tedious.

To quickly verify ISO installers before uploading them to Proxmox storage, I frequently use the ToolCraft Hash Generator. A key advantage is that the SHA-256 or MD5 hashing runs 100% in your browser (client-side). The file is never uploaded to an intermediate server, making it a safe choice for verifying sensitive or proprietary system images.

Stopgap Solutions and Their Practical Shortcomings

Before adopting a dedicated automation tool, our team experimented with two common workarounds:

1. Running Bash Scripts Post-Clone

The initial idea was simple: keep the base template minimal, clone it, and then SSH into the machine to execute a massive bootstrap script that upgrades software packages. While this pulls the latest updates, VM provisioning times ballooned to 10–15 minutes per node. If an upstream package repository suffered network issues or an apt lock got stuck, the entire provisioning pipeline ground to a halt.

2. Relying Solely on Standalone Cloud-Init in Proxmox

Proxmox VE provides seamless native support for Cloud-Init drives, allowing you to pre-inject SSH keys, users, and static networking. However, Cloud-Init is primarily designed for first-boot provisioning (Day 1 configuration). It doesn’t produce an optimized Golden Image with pre-baked Docker engines, specialized runtimes, or kernel-level tunings (such as raising max open file limits or tweaking sysctl parameters).

The Standard Solution: Automating Golden Images with HashiCorp Packer

After more than six months in production, shifting away from manual builds and automating golden images with HashiCorp Packer alongside the proxmox-iso plugin permanently resolved these headaches. The entire VM specification is defined declaratively using clean HCL2 code. When triggered, Packer spins up a temporary VM on Proxmox, serves autoinstall files over a built-in HTTP server, installs the OS and packages, and converts the machine into a template with a single command.

Step 1: Create a Dedicated Proxmox VE API Token

In line with the principle of least privilege, create a scoped API token instead of relying on the root account:

# Create a dedicated role with required permissions for Packer
pveum role add PackerRole -privs "VM.Allocate VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Monitor VM.Audit VM.PowerMgmt Datastore.AllocateSpace Datastore.Audit"

# Create the user and generate an API token
pveum user add packer@pve
pveum user token add packer@pve packer-token -privsep=0
pveum aclmodify / -user packer@pve -role PackerRole

Step 2: Write the Packer Configuration File (HCL)

Create an ubuntu-2204.pkr.hcl file in your project directory. This file specifies hardware resources, the source ISO, and the boot command sent over VNC:

packer {
  required_plugins {
    proxmox = {
      version = ">= 1.1.3"
      source  = "github.com/hashicorp/proxmox"
    }
  }
}

variable "proxmox_api_url" {
  type    = string
  default = "https://192.168.1.100:8006/api2/json"
}

variable "proxmox_api_token_id" {
  type    = string
  default = "packer@pve!packer-token"
}

variable "proxmox_api_token_secret" {
  type      = string
  sensitive = true
  default   = "YOUR_PROXMOX_API_SECRET"
}

source "proxmox-iso" "ubuntu-server" {
  proxmox_url = var.proxmox_api_url
  username     = var.proxmox_api_token_id
  token        = var.proxmox_api_token_secret
  insecure_skip_tls_verify = true

  node                 = "pve-node1"
  vm_id                = 9000
  vm_name              = "ubuntu-2204-golden-template"
  template_description = "Ubuntu 22.04 LTS Golden Image built by Packer"

  iso_file         = "local:iso/ubuntu-22.04.4-live-server-amd64.iso"
  iso_storage_pool = "local"
  unmount_iso      = true

  qemu_agent = true
  cores      = 2
  memory     = 2048
  os         = "l26"

  scsi_controller = "virtio-scsi-pci"

  disks {
    disk_size    = "20G"
    storage_pool = "local-lvm"
    type         = "scsi"
  }

  network_adapters {
    model  = "virtio"
    bridge = "vmbr0"
  }

  # Packer spins up a temporary web server to host user-data
  http_directory = "http"
  boot_command = [
    "<wait>c<wait>",
    "linux /casper/vmlinuz --- autoinstall ds=nocloud-net\\;s=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ <enter><wait>",
    "initrd /casper/initrd <enter><wait>",
    "boot<enter>"
  ]
  boot_wait = "5s"

  ssh_username = "sysadmin"
  ssh_password = "SuperSecurePass123!"
  ssh_timeout  = "20m"
}

Note: Passwords stored in autoinstall configuration files should always use a SHA-512 crypt hash rather than plaintext. When generating high-entropy base passwords before running openssl passwd -6, I find the ToolCraft Password Generator especially helpful. It calculates password entropy directly in the browser without transmitting secrets over the network, making it ideal for creating secure credentials across fresh nodes.

Step 3: Configure Cloud-Init Autoinstall

Create a file named http/user-data. Ubuntu Server’s Subiquity installer reads this configuration to automate disk partitioning, user creation, and package installation:

#cloud-config
autoinstall:
  version: 1
  locale: en_US
  keyboard:
    layout: us
  identity:
    hostname: golden-image
    username: sysadmin
    password: "$6$rounds=4096$salt$hashedpasswordhere..."
  ssh:
    install-server: true
    allow-pw: true
  packages:
    - qemu-guest-agent
    - curl
    - wget
    - git
    - htop
  late-commands:
    - systemctl enable qemu-guest-agent
    - echo "sysadmin ALL=(ALL) NOPASSWD:ALL" > /target/etc/sudoers.d/sysadmin

Step 4: Execute the Automated Build Pipeline

Triggering the template build from your workstation requires just three commands:

# Download required Proxmox plugins
packer init .

# Validate configuration syntax
packer validate .

# Run the automated build pipeline
packer build ubuntu-2204.pkr.hcl

The entire build runs hands-free. First, Packer connects to the Proxmox API to launch a temporary VM with ID 9000. The VM boots from the ISO and fetches the autoinstall files from the lightweight web server spawned by Packer. Once Ubuntu finishes installing, Packer connects via SSH to truncate logs, reset the machine-id, and clean the APT cache (alternatively, you can also inspect and clean virtual disks offline using libguestfs to rescue and modify KVM/Proxmox VMs). Finally, it converts the VM into a production-ready template and shuts it down.

Real-World Impact After 6 Months in Production

Adopting an automated Packer workflow fundamentally transformed how our team manages Proxmox infrastructure:

VM Provisioning Dropped from 20 Minutes to 25 Seconds: When adding new nodes to a cluster, Terraform simply issues a linked clone against VM ID 9000. New virtual machines are fully operational and ready to accept workloads in under half a minute.
Automated Updates via CI/CD: We configured a GitHub Actions workflow that runs every Sunday at 2:00 AM. Our Proxmox Golden Images automatically receive the latest security and kernel patches without any manual effort.
100% Infrastructure Consistency: My personal homelab and our three corporate Proxmox clusters all share the exact same HCL repository. As a result, configuration drift between development and production environments has been virtually eliminated, making it much easier to deploy services or even manage multiple Proxmox clusters centrally.