Packer QEMU performance: speed things up with two-stage builds

Introduction

In the “Packer QEMU: Building Ubuntu 20.04/22.04/24.04 VM images” blog post we briefly mentioned ways to improve Packer QEMU performance. In this article we’ll go into the details on how this is implemented. The key in improving Packer QEMU performance is to split the build into two distinct stages:

  1. Operating system installation stage

  2. Provisioning stage

This setup cuts down development time considerably as repeating the provisioning stage becomes a lot faster.

Our video course about Packer QEMU builder

If you wish to save time you might be interested in our video course “Creating Cloud images from scratch with Packer and QEMU” which covers this topic in much more detail. Additionally you will learn how to build Ubuntu 24.04 and Rocky Linux 9 VM images from scratch. Moreover, you will learn how to import them into AWS and Azure. The course students also get access to a Git repository with tested, ready-to-use code. Overall this can simplify the build and publishing process dramatically for you.

The first stage build: automate the operating system installation

The first build stage is responsible for installing the operating system. Here’s an example for Ubuntu 24.04:

# os-install.pkr.hcl
packer {
  required_version = "= 1.11.1"
  required_plugins {
    qemu = {
      version = "= 1.1.0"
      source = "github.com/hashicorp/qemu"
    }
  }
}

source "qemu" "iso" {
  vm_name              = "ubuntu-2404-amd64.raw"
  iso_url              = "https://www.releases.ubuntu.com/24.04/ubuntu-24.04-live-server-amd64.iso"
  iso_checksum         = "8762f7e74e4d64d72fceb5f70682e6b069932deedb4949c6975d0f0fe0a91be3"
  memory               = 1500
  disk_image           = false
  output_directory     = "build/os-base"
  accelerator          = "kvm"
  disk_size            = "12000M"
  disk_interface       = "virtio"
  format               = "raw"
  net_device           = "virtio-net"
  boot_wait            = "3s"
  boot_command         = [
    "ewait",
    "downdowndownend",
    " autoinstall ds=\"nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/\" ",
    "f10"
    ]
  http_directory       = "http"
  shutdown_command     = "echo 'packer' | sudo -S shutdown -P now"
  ssh_username         = "packer"
  ssh_password         = "packer"
  ssh_timeout          = "60m"
}

build {
  name    = "iso"
  sources = ["source.qemu.iso"]
}

In this case the http directory defined by http_directory parameter must contain meta-data and user-data files; for details see our Packer QEMU blog post. Note how the disk_image parameter is set to false so that Packer knows it’s dealing with an ISO.

The second stage build: run provisioning

The second build stage is what improves Packer QEMU performance. Instead of rerunning the operating system installation every time you run Packer, you run it once. Thereafter you just run the Packer provisioners on top of the disk image build in the previous stage.

Although it is possible to combine both build stages into one Packerfile, it is cleaner to keep the build stages in separate files. Here’s a simple example for Ubuntu 24.04 and AWS:

# aws.pkr.hcl
packer {
  required_version = "= 1.11.1"
  required_plugins {
    qemu = {
      version = "= 1.1.0"
      source = "github.com/hashicorp/qemu"
    }
  }
}
source "qemu" "base-img" {
  vm_name           = "ubuntu-2404-amd64.raw"
  iso_url           = "build/os-base/ubuntu-2404-amd64.raw"
  iso_checksum      = "none"
  disk_image        = true
  memory            = 1500
  output_directory  = "build/aws"
  accelerator       = "kvm"
  disk_size         = "12000M"
  disk_interface    = "virtio"
  format            = "raw"
  net_device        = "virtio-net"
  boot_wait         = "3s"
  shutdown_command  = "echo 'packer' | sudo -S shutdown -P now"
  ssh_username      = "packer"
  ssh_password      = "packer"
  ssh_timeout       = "60m"
  headless          = true
}

build {
  name = "aws"

  sources = ["source.qemu.base-img"]

  provisioner "shell" {
    script            = "../shared/debian/dist_upgrade.sh"
    execute_command   = "echo 'packer' | sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
  }
}

Note how disk_image is now set to true. Due to this Packer will launch a virtual machine from the disk image. It will then run provisioners and wrap the results into a new VM up as a new disk image.

Running the builds

The build process is surprisingly straightforward. To install the operating system:

packer build -force os-install.pkr.hcl

Once the operating system installation stage does what you want, you can run the provisioning stage:

packer build -force aws.pkr.hcl

Whenever you make changes to your provisioning scripts you just run the provisioning stage again.

Multi-cloud support

Different Clouds such as AWS and Azure require very different provisioning steps. For the most part you only need to install the operating system once and then separate the second stage builds into different files such as aws.pkr.hcl and azure.pkr.hcl.