Notes about puppet-litmus testing

Puppet Litmus is a Puppet acceptance test harness that leverages on many existing, proven technologies. In the long run Litmus aims to replace Beaker. However, the latter is currently still the only way to run multi-machine (e.g. cluster) acceptance tests in a standardized way. For example we use Beaker for our Keycloak domain mode cluster tests in puppet-module-keycloak. On a high-level the test process is the same as with Beaker:

• Provision the SUT (“System Under Test”)

• Run tests

• Tear down the SUT

Litmus supports various backends for creating the SUT, including, but not limited to, Docker and Vagrant. Provisioning of the SUT is done with Puppet Bolt. Acceptance tests are written using ServerSpec.

To get started with Litmus testing check out the following articles first:

• Official documentation. It is sparse, but gives you at least a rough idea of what you need to do and how Litmus works.

• Acceptance testing Puppet modules with PDK, Litmus and Vagrant

• Integrate Litmus acceptance tests into CD for PE

In a perfect world where everything is set up correctly the Litmus test process in Vagrant is as follows:

# Create the SUT. "vagrant" is an arbitrary name for the
# node list in provision.yaml
pdk bundle exec rake 'litmus:provision_list[vagrant]'

# Install Puppet 6 on the SUT
pdk bundle exec rake 'litmus:install_agent[puppet6]'

# This step is only required with Vagrant
pdk bundle exec bolt task run provision::fix_secure_path --modulepath spec/fixtures/modules -i inventory.yaml -t ssh_nodes

# Install our module and its dependencies on the SUT
pdk bundle exec rake "litmus:install_module"

# Run the acceptance tests
pdk bundle exec rake litmus:acceptance:parallel

# Tear down the SUT
pdk bundle exec rake litmus:tear_down

As the world is rarely perfect, this will probably break on many levels. Here are the issues I encounted - your mileage may vary:

• You can’t have “StrictHostKeyChecking no” in ~/.ssh/config or Litmus will choke with a somewhat misleading error message.

• Litmus uses puppet module install to install the dependency modules. on top of that it gets the dependency module list from metadata.json and not from .fixtures.yml or Puppetfile. This introduces some challenges:

  • metadata.json does not support locking Puppet module dependencies to certain versions (e.g. puppetlabs/stdlib = 7.0.0). Therefore you will need to give an upper and lower bound for the module versions. This means that your unit tests, which use .fixtures.yml might pass, but you Litmus acceptance tests might fail, because a dependency module version is not identical to the one used in the unit tests. Or the other way around.

  • puppet module install downloads module dependencies by default. This canpotentially result in a dependency hell. It seems possible to override dependency management, though.

  • For Puppet Litmus you’d prefer to have strict dependencies (in metadata.json), but for generic use you’d like to have loose dependencies to avoid creating a depency hell for users of your module who may be using puppet module install.

Some useful tricks I learned while debugging Litmus:

# Run arbitrary command on the SUT
pdk bundle exec bolt command run "ls /etc/puppetlabs/code/environments/production/modules" -i inventory.yaml -t ssh_nodes

# List all Litmus (rake) tasks
pdk bundle exec rake --tasks

# Debug dependency module installation
pdk bundle exec rake "litmus:install_module" --trace

More blog posts about Litmus will follow as I figure out reasonable solutions to the challenges listed above.

More on Puppet testing:

• Introduction

• Puppet