Two-stage Qemu builds with Packer

In the Building Ubuntu 20.04 qemu images with Packer blog post we briefly touched on the topic of two-stage builds with Packer to save time when working on the provisioning scripts. In that article the setup was ad hoc and was based on having two separate Packerfiles (*.pkr.hcl). It is, however, possible to have a two-stage build implemented using a single Packer project. Here's an example:

# Install the operating system but do not run any provisioning scripts
source "qemu" "iso" {
  vm_name              = "ubuntu-2004-amd64-qemu-iso-build"
  iso_url              = "http://www.releases.ubuntu.com/20.04/ubuntu-20.04.2-live-server-amd64.iso"
  iso_checksum         = "sha256:d1f2bf834bbe9bb43faf16f9be992a6f3935e65be0edece1dee2aa6eb1767423"
  memory               = 1280
  disk_image           = false
  output_directory     = "ubuntu-2004-amd64-qemu-iso-output"
  accelerator          = "kvm"
  disk_size            = "5000M"
  disk_interface       = "virtio"
  format               = "qcow2"
  net_device           = "virtio-net"
  boot_wait            = "3s"
  # Inspired by https://nickhowell.uk/2020/05/01/Automating-Ubuntu2004-Images/
  boot_command         = [
    # Make the language selector appear...
    " <up><wait>",
    # ...then get rid of it
    " <up><wait><esc><wait>",

    # Go to the other installation options menu and leave it
    "<f6><wait><esc><wait>",

    # Remove the kernel command-line that already exists
    "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
    "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
    "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",

    # Add kernel command-line and start install
    "/casper/vmlinuz ",
    "initrd=/casper/initrd ",
    "autoinstall ",
    "ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/ubuntu-2004-amd64-qemu/ ",
    "<enter>"
    ]
  http_directory       = "ubuntu-2004-amd64-qemu/http"
  shutdown_command     = "echo 'packer' | sudo -S shutdown -P now"
  # These are required or Packer will panic, even if no provisioners are not configured
  ssh_username         = "openvpnas"
  ssh_password         = "ubuntu"
  ssh_timeout          = "60m"
}

# Run provisioning scripts on top of the image created above
source "qemu" "img" {
  vm_name           = "ubuntu-2004-amd64-qemu-img-build"
  iso_url           = "ubuntu-2004-amd64-qemu-iso-output/ubuntu-2004-amd64-qemu-iso-build"
  # Checking the checksum would be pointless as we just built the source image
  iso_checksum      = "none"
  disk_image        = true
  memory            = 1280
  output_directory  = "ubuntu-2004-amd64-qemu-img-output"
  accelerator       = "kvm"
  disk_size         = "5000M"
  disk_interface    = "virtio"
  format            = "qcow2"
  net_device        = "virtio-net"
  boot_wait         = "3s"
  shutdown_command  = "echo 'packer' | sudo -S shutdown -P now"
  ssh_username      = "openvpnas"
  ssh_password      = "ubuntu"
  ssh_timeout       = "60m"
}

build {
  name = "iso"

  sources = ["source.qemu.iso"]
}

build {
  name = "img"

  sources = ["source.qemu.img"]

  provisioner "file" {
    sources     = [ "ubuntu-2004-amd64-qemu/provision/somefile.sh" ]
    destination = "/tmp/"
  }

  provisioner "shell" {
    script          = "ubuntu-2004-amd64-qemu/provision/script.sh"
    execute_command = "sudo bash {{.Path}}"
  }
}

Packer always tries to launch all builds in parallel, which makes a lot of sense in the Cloud context. For example you might be building for AWS and Azure at the same time.

With two-stage Qemu builds it means that Packer will fall flat on its face because the second phase launches before the first one has finished. So, we need to explicitly run the stages in order:

packer build -only=img.* ubuntu-2004-amd64-qemu

It is assumed that the *.pkr.hcl files are in directory called "ubuntu-2004-amd64-qemu". Once stage one is finished you can start the second stage:

packer build -only=iso.* ubuntu-2004-amd64-qemu

In the above case the full paths to the two builds would be iso.qemu.iso and img.qemu.img, respectively, but we used a wildcard. The path syntax is kind of explained in Packer documentation, but it still feels quite weird.