Building Ubuntu 20.04 qemu images with Packer

July 5, 2021 

Introduction

We use Packer a lot, but I had not so far generated any Qemu images with it. This was a fun project because it allowed (or forced) me to learn autoinstall, Ubuntu's cloud-init style installation automation, which by the way easier to work with than Debian-style preseeds. For more information on preseeding for pre-20.04 Ubuntu and Debian see our Ubuntu and Debian preseeding tips blog post. Now to the actual topic. The impatient can skip to the very end to find the full HCL2-formatted Packerfile and user-data (autoinstall) file.

Packer has official support for Qemu builders. There are two main ways to build Ubuntu 20.04 qemu images:

  • Start from an installation ISO: do a true hands-free automated installation with autoinstall.
  • Start from an Ubuntu Cloud image: modify an existing disk image to suit your needs without using autoinstall

The former provides the most flexibility in how things are setup, so that's the approach described in this article.

To be able to build Qemu images with Packer you have to solve several challenges:

  • Ensuring that the local user used to run Packer is able to launch kvm/qemu virtual machines
  • Making sure that Packer uses ISO instead of a disk image
  • Make sure the installer has enough memory (1GB) to run
  • Starting the Ubuntu installer without prompts
  • Creating an autoinstall file
  • Sharing the autoinstall file to the Ubuntu installer
  • Pointing the Ubuntu installer to the autoinstall (user-data, meta-data) files
  • Ensuring that Packer's SSH provisioners are able to work

The examples below use the new HCL2 syntax for Packer.

Getting Packer to boot the installer

Many Linux distributions come with libvirt and qemu installed out of the box. To allow Packer to create VMs and images all you typically need to do is:

usermod -a -G libvirt your_username

To ensure that Packer knows you want to launch an ISO-based installation ensure that you have this in your builder configuration:

disk_image = false

This is the default, so in most cases you can just omit it.

Next ensure that the VM Packer creates has enough memory to work. With the default of 512MB you will get strange errors almost immediately when the installer starts up:

memory = 1024

To get the installer to start without prompts Packer actually sends keystrokes over VNC, just like you would if you ran the installer interactively. For this reason the examples you get from the Internet will not work if the installation menu has changed in the meanwhile. On the other hand figuring out the correct keystrokes is easy as you can test the process manually. First you need to wait a bit before letting Packer to start typing:

boot_wait = "3s"

The correct keystrokes In my case (Ubuntu 20.04 server as of 5th July 2021) were:

boot_command      = [
  # Make the language selector appear...
  " <up><wait>",
  # ...then get rid of it
  " <up><wait><esc><wait>",

  # Go to the other installation options menu and leave it
  "<f6><wait><esc><wait>",

  # Remove the kernel command-line that already exists
  "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>"<bs><bs>",
  "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
  "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",

  # Add kernel command-line and start install
  "/casper/vmlinuz ",
  "initrd=/casper/initrd ",
  "autoinstall ",
  "ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/ubuntu-2004-amd64-qemu/ ",
  "<enter>"
]

The "ds" line tells the Ubuntu installer where to find the HTTP server serving user-data and meta-data. In your Packerfile you need to define the webroot for the web server Packer launches:

http_directory = "http-server"

You need to have a corresponding directory next to your Packerfile. With the above "ds" line you'd do

mkdir -p http-server/ubuntu-2004-amd64-qemu

Now, when you add an empty meta-data file as well as user-data to that directory the Ubuntu installer will find and use them. See the Providing the autoinstall config section in Ubuntu's official documentation for more details on how this works. The parameters in curly braces are automatically populated by Packer, so you can just copy them as-is.

The above things shold get you through the install. But Packer's provisioning might not work out of the box. In your Packerfile you need to configure SSH:

ssh_username = "ubuntu"
ssh_password = "ubuntu"
ssh_timeout  = "60m"

Creating the user-data file

First we edit user-data to ensure that "ubuntu" user is present and is able to sudo with a password:

identity:
  hostname: ubuntu
  username: ubuntu
  password: $6$yV./p0CXmpUSsU1c$gXKXP/7hpiIyQKEl0yV5OH9/82vXQ9QdPz2heaf6fPA6OOuh1aERzzCv2573x0tVU/Rx7ZjIk6M022AFRZDf2.

late-commands:
  - "echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/ubuntu"
  - "chmod 440 /target/etc/sudoers.d/ubuntu"

The password is encrypted and salted version of "ubuntu" for the sake of example, so don't use it in production (pretty please). You can create a hash by various means, but creating a new user on an existing system and getting the hash from /etc/shadow as root is a good bet.

We also need some SSH configuration in user-data. First you should ensure that SSH server is installed and enabled and allows password logins:

ssh:
  allow-pw: true
  authorized-keys: []
  install-server: true

However, you also should stop the ssh service in the installer to prevent Packer from trying to connect to the Ubuntu installer via SSH, thinking it can start the provisioning process:

  early-commands:
    - systemctl stop ssh

This should be enough to make building Qemu images with Packer and provisioning work.

Here's an example Packerfile named descriptively ubuntu-2004-amd64-qemu.pkr.hcl:

source "qemu" "ubuntu-2004-amd64-qemu" {
  vm_name           = "ubuntu-2004-amd64-qemu-build"
  iso_url           = "http://www.releases.ubuntu.com/20.04/ubuntu-20.04.2-live-server-amd64.iso"
  iso_checksum      = "sha256:d1f2bf834bbe9bb43faf16f9be992a6f3935e65be0edece1dee2aa6eb1767423"
  memory            = 1024
  disk_image        = false
  output_directory  = "output-ubuntu-2004-amd64-qemu"
  accelerator       = "kvm"
  disk_size         = "5000M"
  disk_interface    = "virtio"
  format            = "qcow2"
  net_device        = "virtio-net"
  boot_wait         = "3s"
  boot_command      = [
    # Make the language selector appear...
    " <up><wait>",
    # ...then get rid of it
    " <up><wait><esc><wait>",

    # Go to the other installation options menu and leave it
    "<f6><wait><esc><wait>",

    # Remove the kernel command-line that already exists
    "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
    "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",
    "<bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs><bs>",

    # Add kernel command-line and start install
    "/casper/vmlinuz ",
    "initrd=/casper/initrd ",
    "autoinstall ",
    "ds=nocloud-net;s=http://{{.HTTPIP}}:{{.HTTPPort}}/ubuntu-2004-amd64-qemu/ ",
    "<enter>"
    ]
  http_directory    = "http-server"
  shutdown_command  = "echo 'packer' | sudo -S shutdown -P now"
  ssh_username      = "ubuntu"
  ssh_password      = "ubuntu"
  ssh_timeout       = "60m"
}

build {
  sources = ["source.qemu.ubuntu-2004-amd64-qemu"]

  provisioner "file" {
    sources     = [ "provisioning/first-config",
                    "provisioning/second-config"]
    destination = "/home/ubuntu/"
  }

  provisioner "shell" {
    script          = "provisioning/init.sh"
    execute_command = "sudo bash {{.Path}}"
  }
}

And here's an example of the user-data (autoinstall) file:

#cloud-config
autoinstall:
  version: 1
  early-commands:
    - systemctl stop ssh
  locale: en_US.UTF-8
  keyboard:
    layout: us
  apt:
    preserve_sources_list: false
    primary:
        - arches: [i386, amd64]
          uri: "http://archive.ubuntu.com/ubuntu"
        - arches: [default]
          uri: "http://ports.ubuntu.com/ubuntu-ports"
    geoip: true
  storage:
    layout:
      name: lvm
  identity:
    hostname: ubuntu
    username: ubuntu
    password: $6$yV./p0CXmpUSsU1c$gXKXP/7hpiIyQKEl0yV5OH9/82vXQ9QdPz2heaf6fPA6OOuh1aERzzCv2573x0tVU/Rx7ZjIk6M022AFRZDf2.
  ssh:
    allow-pw: true
    authorized-keys: []
    install-server: true
  late-commands:
    - "echo 'ubuntu ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/ubuntu"
    - "chmod 440 /target/etc/sudoers.d/ubuntu"

As mentioned above, the meta-data file can be empty.

Launching the build

Assuming your system is set up correctly you can build with just:

packer build

If you split your Packer configuration into multiple .pkr.hcl file you need to place all the files into a directory and use

packer build <directory>

Tips

Using custom cloud-init configuration with autoinstall

While autoinstall uses a yaml-based cloud-init style syntax, you only use parameters supported by autoinstall in it. If you need cloud-init's more advanced feature you need to pass a user-data section to it. Here's a simple example that shows how to enable the root user with highly insecure password and to set the timezone:

  user-data:
    timezone: Europe/Helsinki
    disable_root: false
    chpasswd:
      list: |
        root:secret

Anything you can do with cloud-init should be possible to do from the user-data section. Note that these settings will get applied on first boot, so if you mount the disk image created by Packer you might not see them. If you create a VM from the disk image you should notice how cloud-init applies these changes.

Mounting qcow2 images

Mounting the qcow2 images created by Packer can be very useful when debugging autoinstall. You need to have libguestfs-tools or equivalent package installed for this to work.

The basic process (as root) is this:

LIBGUESTFS_BACKEND=direct guestmount -a output/ubuntu-2004-amd64-qemu-build -m /dev/sda2 /mnt/guestmount/

If you give guestmount an invalid partition it will helpfullly print out a list of valid partition names:

guestmount: ‘/dev/sda3’ could not be mounted.
guestmount: Did you mean to mount one of these filesystems?
guestmount:     /dev/sda1 (unknown)
guestmount:     /dev/sda2 (ext4)
guestmount:     /dev/ubuntu-vg/ubuntu-lv (ext4)

Note how guestmount supports logical volumes as well. Quite nice, compared to the older and hackier methods I've used in the past.

To unmount:

umount /mnt/guestmount

Note that mounting the volume may change the image checksum. This is not normally a problem, but if you want to save time by using two-stage builds (see below) you should be aware of it.

Using two-stage builds to save time

Provisioning operating systems is a very slow process and autoinstall is no exception. In the Packer context you're doing two things:

  1. Automating the installation of the operating system
  2. Running provisioning scripts

It gets tedious to go through phase 1 when you're just trying to make your provisioning scripts work - with every typo you end up wasting about 10 minutes of your time. To alleviate this problem you can split your Packer builds into two, at least until your provisioning scripts work as expected.

Your phase 1 Packerfile should create the base operating system image without running any provisioners on it. Phase 2 Packerfile will then use the phase 1-generated disk image as a source and run your provisioners on it. The key settings in phase 2 Packerfile are these:

  # Use different VM name (safety precaution)
  vm_name = "ubuntu-2004-amd64-qemu-build-from-image"

  # Point the build to the disk image created in phase 1
  iso_url = "/path/to/ubuntu-2004-amd64-qemu-build"

  # You don't want to be updating the checksum every time you rebuild phase 1
  iso_checksum = "none"

  # Make Packer realize we're building from a disk image
  disk_image = true

If you want to make two-stage builds your de facto standard please have a look at our Two-stage Qemu builds with Packer blog post.

For further information please refer to the official documentation:

These external web pages were used as a source:

Samuli Seppänen
Samuli Seppänen
Author archive
menucross-circle