Serverless Puppet with control repo, Hiera, roles and profiles and Puppet Bolt

The traditional way of managing systems with Puppet is to install Puppet agent on the nodes being managed and point those agents to a Puppet server (more details here). This approach works well for environments with tens or hundreds of nodes, but is an overkill for small environments with just a handful of nodes. Fortunately nowadays Puppet Bolt enables you to push configurations to the managed nodes via SSH and WinRM. When using Bolt the catalog (desired state) is created on the computer running Puppet Bolt using facts from managed nodes and then sent to the managed nodes to be applied.

While the configuration management use-case with Puppet Bolt is clearly supported, there is not much documentation available on how to use it with the well-tested control repository paradigm, so this article aims to fill that void. I've written two articles that touched on this topic earlier, but as Puppet Bolt develops really fast they have already become more or less outdated:

• Puppet Bolt with a Puppet control repository
• Using Puppet Bolt to apply roles to nodes, part 1

Let's start with the layout of the control repository which is very typical:

.
├── bolt-project.yaml
├── data
│   ├── common.eyaml
│   ├── common.yaml
│   └── nodes
│       └── mynode.yaml
├── hiera.yaml
├── inventory.yaml
├── keys
│   ├── private_key.pkcs7.pem
│   └── public_key.pkcs7.pem
├── plans
│   └── configure.pp
├── README.md
└── site
    ├── profile
    │   └── manifests
    |       └── unixbase.pp
    └── role
        └── manifests
            └── generic.pp

The first thing you need is bolt-project.yaml. We'll go through it in pieces:

---
name: acme
concurrency: 3
format: human

A simple project file like this works well if you're not using Puppet modules, e.g. just doing tasks or using plans to orchestrations. However, when using roles and profiles you probably want to place them to the site subdirectory, so you need to add that to the modulepath:

modulepath:
  - site

Bolt's default modules directory is .modules and that one does not have to added to modulepath explicitly.

The final thing you need in bolt-project.yaml is definition of external Puppet modules you wish to use. For reasons unknown Puppet Bolt nowadays uses bolt-project.yaml to define module dependencies. Bolt uses that information to download the modules and to dynamically manage a Puppetfile with equivalent contents. This does not work particularly well in an environment where some nodes are managed by a Puppet server + agents and some by Puppet Bolt, but in this article's use-case that does not matter. The module dependency definition in bolt-project.yaml looks like this:

modules:
  - name: puppetlabs/concat
    version_requirement: '7.1.1'
    resolve: false
  - name: puppetlabs/firewall
    version_requirement: '3.3.0'
    resolve: false
  - name: puppetfinland-packetfilter
    git: 'https://github.com/Puppet-Finland/puppet-packetfilter.git'
    ref: 'cb3ca18ebbce594bd7e527999390c0137daf3a57'
    resolve: false
  - name: puppetlabs/stdlib
    version_requirement: '8.1.0'
    resolve: false
  - name: saz-timezone
    version_requirement: '6.1.0'
    resolve: false

Note how resolve: false is set for every module. This is based on bad experiences (with librarian-puppet) with automatic dependency resolution in conjunction with Puppet modules. To be more exact the dependencies defined in Puppet modules' metadata.json are not usually up-to-date or even correct, which will inevitably lead into dependency conflicts you need to work your way around. For details on the modules setting syntax see the official documentation.

Once you've created bolt-project.yaml it's time to move on to another import file, inventory.yaml. Here's a really simplistic sample:

targets:
  - uri: '62.101.209.12'
    name: mynode
    config:
      transport: ssh
config:
  ssh:
    host-key-check: false
    user: root

It defines just one target to connect to using an IPv4 addrress and gives it a symbolic name ("mynode") that can be used in Puppet Bolt command-lines when defining targets. On top of that some defaults are set for SSH connections. This kind of static inventory works ok if you only have a handful of nodes, especially when you group your targets. In bigger environments you probably want to use inventory plugins to create inventories dynamically instead of having to keep them up to date yourself.

The next step is to configure hiera.yaml because we want to separate code from data. Here's a very simple version that supports decrypting secrets with hiera-eyaml:

---
version: 5
defaults:
 datadir: data
hierarchy:
 - name: "Sensitive data (passwords and such)"
   lookup_key: eyaml_lookup_key
   paths:
     - "nodes/%{trusted.certname}.eyaml"
     - "common.eyaml"
   options:
     pkcs7_private_key: keys/private_key.pkcs7.pem
     pkcs7_public_key:  keys/public_key.pkcs7.pem
 - name: "Normal data"
   data_hash: yaml_data
   paths:
     - "nodes/%{trusted.certname}.yaml"
     - "common.yaml"

Make sure you copy your eyaml keys in keys directory in Bolt project root.

Now you can set up your profiles and roles. Here's an example of extremely minimalistic profile (site/profile/manifests/unixbase.pp):

#
# @summary base classes included on all nodes
#
class profile::unixbase {
  $timezone = lookup(profile::unixbase::timezone, String)

  class { '::timezone':
    timezone => $timezone,
  }
}

For this to work you need to have data in data/common.yaml:

profile::unixbase::timezone: 'Etc/UTC'

If you have any secret data you can put it to eyaml files just like you would normally.

Next you need a role that includes the unixbase profile. Here we use a generic role (site/role/manifests/generic.pp):

#
# @summary dummy role used for demonstration purposes
#
class role::generic {
  contain profile::unixbase
}

In a real environment you'd include profiles that actually configure the node to do something useful.

Next our node (mynode) needs to include role::generic. With the hiera.yaml defined above you'd add data/nodes/mynode.yaml with contents like this:

classes:
  - role::generic

In a Puppet server + agent architecture this would be all you need to do. However, in case of Puppet Bolt you need to one more step: add a Bolt plan (plans/configure.pp) that glues everything together:

plan acme::configure (
  TargetSpec $targets,
  Boolean    $noop = false
) {
  # Install Puppet Agent
  $targets.apply_prep

  # Run Puppet on several targets in parallel
  $apply_result = apply($targets, '_noop' => $noop, '_catch_errors' => true) {
    $classes = lookup('classes', Optional[Array[String]], 'first', undef)
    if $classes {
      $classes.each |$c| {
        include $c
      }
    }
  }

  # Sometimes Puppet run may fail so early that there's no output, for
  # example if a suitable provider can't be found for a resource.
  # In that case you can uncomment this to print the apply result and
  # figure out what went wrong. 
  #out::message($apply_result)

As you can see, there's no need to create separate Bolt plans for each role.

Now, before you can apply any code with Puppet you need to download the Puppet modules defined in bolt-project.yaml:

bolt module install --force

The --force option tends tends to be required as Bolt apparently tries to somehow co-exist with Puppetfiles that are manually managed.

Now you run the plan to configure you nodes:

bolt plan run -v acme::configure -t all

The -v option enables you to see what changes Puppet makes on the target nodes.

Finally you should configure .gitignore so that you don't accidentally version things like eyaml keys:

/keys
.modules
.plan_cache.json
.resource_types
.task_cache.json
Puppetfile
bolt-debug.log
.rerun.json

Note how Puppetfile is not versioned: as mentioned above Puppet Bolt manages it dynamically and we don't need/want to version it.