Debugging Packer provisioning
Packer is an excellent tool for automating creation of machine images such as AWS AMIs or Docker container images. This is particularly useful when you need to scale up by spinning up instances on-demand or if you want to package software that is challenging to manage with a configuration management system. It is also extremely useful when creating actual packaged product for AWS Marketplace, for example.
In Packer you take a “base box” such as Ubuntu 18.04 base image and modify its contents with provisioning, for example to install and configure software on the image. Several different types of provisioners are supported, including but not limited to shell scripts, Puppet manifests and Ansible playbooks. The concept of provisioners is the same as in Vagrant and Terraform.
As with any provisioning most of the time you’re just waiting for the machine - often your own laptop - to run through the provisioning steps. The challenge is that you build up the provisioning steps one by one, and if they fail, you need to start from scratch. Also, the failure generally occurs at the last step (unless you’ve just done some refactoring) so the more steps you have, the more you have to wait between tries.
Fortunately Packer has two mechanisms which help you debug issues without going through the full image build process after every try. The first one is the --debug option, which allows you to run provisioning interactively step-by-step and going in to the temporary cloud instance to see why a certain step failed. Alternatively you can use -on-error=abort to keep the build artifacts (i.e. temporary cloud instance) if provisioning fails. Then you can SSH in to the instance and see what failed and figure out a solution.