Testing Puppet feature environments with Puppet Bolt

Puppet feature environments are an excellent way to test code before deploying it, typically to production. They allow testing Puppet runs on no-operation mode across the whole node population managed by Puppet. There are sometimes cases where your code changes potentially impact many nodes and you're not exactly sure of their scope or effect. In that case it helps to test your code on all Puppet-managed nodes. The challenge is that Puppet open source does not have any built-in mechanism to do that. This is where we can make good use of Puppet Bolt.

The first part of the job is to deploy the feature branch as a Puppet environment. Here we use a Bolt task which can be used independently or as part of our feature branch testing plan. Bolt tasks can be placed into Puppet modules, but here we place it to <bolt-project-root>/tasks. The first part of the task is a metadata file, here tasks/deploy_feature_branch.json. Its purpose is to define what parameters are passed to the script, what their data types should be and how they're passed to the script.

{
  "description": "Deploy a Puppet feature environment with r10k",
  "input_method": "environment",
  "parameters": {
    "branch": {
      "description": "git branch to deploy as puppet environment",
      "type": "String"
    }
  }
}

As we can see, the "branch" parameter is passed to the script via an environment variable. We set the datatype to String to prevent passing something silly as the branch name (e.g. Array or a Boolean).

The script lives in <bolt-project-root< tasks directory and has the same name as the metadata file, except for the file extension. In this case it is called deploy_feature_branch.sh and it looks like this:

#!/bin/sh
#
BRANCH=$PT_branch

if [ "$BRANCH" = "" ]; then
    echo "ERROR: branch name missing!"
    exit 1
fi

/opt/puppetlabs/puppet/bin/r10k deploy environment $BRANCH -vp 2>&1
/opt/puppetlabs/bin/puppet generate types --environment $BRANCH --force

When the task is called, Bolt creates a new environment variable PT_branch because the task metadata has this is in:

"input_method": "environment",

PT_branch gets its value from the branch parameter passed on the Bolt command-line.

Running this r10k deployment task independently is easy:

$ bolt task run myproject::deploy_feature_branch branch=some_branch --run-as root -t puppet.example.org

The "myproject" part should match your Bolt project name.

The second part of the challenge is to create a Bolt plan that first deploys the feature branch and then runs Puppet in no-operation mode against some or all nodes in the Bolt inventory. Plans don't have separate metadata as they are written in Puppet language which has metadata support built-in. Like tasks, plans can be placed into Puppet modules, but here we place the plan to <bolt-project-root>/plans/test_feature_branch.pp:

#
# @summary deploy a feature branch with r10k and test it
#
plan myproject::test_feature_branch
(
  String     $branch,
  TargetSpec $puppetserver = 'puppet.example.org',
  TargetSpec $targets = 'all'
)
{
  $dfb = run_task('myproject::deploy_feature_branch', $puppetserver, { 'branch' => $branch })

  out::message($dfb.to_data[0]['value']['_output'])

  # Convert the result into a data structure we can query
  $dfbd = $dfb.to_data[0]

  unless $dfbd['status'] == 'success' {
    fail('ERROR: failed to deploy branch as a Puppet environment!')
  }

  $pr = run_command("/opt/puppetlabs/bin/puppet agent --onetime --verbose --show_diff --no-daemonize --color=true --environment ${branch} --noop", $targets)

  $pr.to_data.each |$result| {
    out::message("Output from ${result['target']}")
    out::message($result['value']['stdout'])
  }
}

To understand what is happening here requires some basic understanding of how Bolt plan functions work. When a function such as run_task or run_command runs it will actually return a ResultSet object which contains some metadata and output from that function. For example our deploy_feature_branch task produces ResultSet like this:

[
  {
    "target": "puppet.example.org",
    "action": "task",
    "object": "myproject::deploy_feature_branch",
    "status": "success",
    "value": {
        "_output": "INFO\t -> Using Puppetfile '/etc/puppetlabs/code/environments/first_env/Puppetfile'\nINFO\t -> Using Puppetfile '/etc/puppetlabs/code/environments/second_env/Puppetfile'\n --- snip ---"
  }
]

The ResultSet object is a bit special: if you print it with out::message function you might be fooled into thinking that it's already a Hash. However, it actually has to be converted to a Hash with the to_data function before any of the values in it can be referenced. Note how it is a set - this is because the same function (e.g. run_task or run_command) could run on multiple targets. If you know that there was only one target, you can just reference the results with

$result = $resultset.to_data[0]

The ResultSet returned by run_command function is similar:

[
  {
    "target": "server.example.org",
    "action": "command",
    "object": "/opt/puppetlabs/bin/puppet agent --onetime --verbose --show_diff --no-daemonize --color=true --environment second_env --noop",
    "status": "success",
    "value": {
      "stdout": "\u001b[0;32mInfo: Using configured environment 'beta_20210618'\u001b[0m\n\u001b[0;32mInfo: Retrieving pluginfacts\u001b[0m\n\u001b[0;32mInfo: Retrieving plugin\u001b[0m\n\u001b[0;32mInfo: Retrieving locales\u001b[0m\n\u001b[0;32mInfo: Loading facts\u001b[0m\n\u001b[0;32mInfo: Applying configuration version '1624024558'\u001b[0m\n\u001b[mNotice: Class[Apt::Update]: Would have triggered 'refresh' from 1 event\u001b[0m\n\u001b[mNotice: Stage[main]: Would have triggered 'refresh' from 1 event\u001b[0m\n\u001b[mNotice: Applied catalog in 1.24 seconds\u001b[0m\n",
      "stderr": "",
      "merged_output": "\u001b[0;32mInfo: Using configured environment 'beta_20210618'\u001b[0m\n\u001b[0;32mInfo: Retrieving pluginfacts\u001b[0m\n\u001b[0;32mInfo: Retrieving plugin\u001b[0m\n\u001b[0;32mInfo: Retrieving locales\u001b[0m\n\u001b[0;32mInfo: Loading facts\u001b[0m\n\u001b[0;32mInfo: Applying configuration version '1624024558'\u001b[0m\n\u001b[mNotice: Class[Apt::Update]: Would have triggered 'refresh' from 1 event\u001b[0m\n\u001b[mNotice: Stage[main]: Would have triggered 'refresh' from 1 event\u001b[0m\n\u001b[mNotice: Applied catalog in 1.24 seconds\u001b[0m\n",
      "exit_code": 0
    }
  }
]

The raw ResultSets tend to look very nasty, which is why we loop through all the objects in them and just print their standard output. For example:

  $pr.to_data.each |$result| {
    out::message("Output from ${result['target']}")
    out::message($result['value']['stdout'])
  }

This produces very nice, human-readable output. The output actually has colors as well, but those are not shown here:

Output from server.example.org
Info: Using configured environment 'second_env'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Applying configuration version '1624025340'
Notice: Class[Apt::Update]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 1.18 seconds
Finished: plan myproject::test_feature_branch in 40.96 sec
Plan completed successfully with no result

So, to test the feature branch on all nodes in the inventory:

$ bolt plan run myproject::test_feature_branch branch=second_env targets=all --run-as root

To test it on a subset of nodes:

$ bolt plan run myproject::test_feature_branch branch=second_env targets=server.example.org,www.example.org --run-as root