Limiting the scope of puppet-rspec tests

January 22, 2021 – Samuli Seppänen

Some Puppet modules like puppet-module-keycloak have hundreds of unit tests. That is good for test coverage, but waiting for test results hurts your productivity when you're developing tests for your new code. There are at least two ways to (temporarily) limit the scope of the tests that you run. First method is baked in into the Puppet Development Kit command-line:

$ pdk test unit --tests=spec/classes/init_spec.rb

This runs only the tests in the specified file.

Puppet rspec tests are often run on all supported platforms defined in metadata.json. That can be easily done with help from rspec-puppet-facts:

  on_supported_os.each do |os, facts|
    context "on #{os}" do
      --- snip ---
    end
  end

This pattern is convenient, but increases test count greatly. Fortunately you can modify the spec file to only run tests on a single platform:

 bionic = { supported_os: [{ 'operatingsystem' => 'Ubuntu', 'operatingsystemrelease' => ['18.04'] }] }

  on_supported_os(bionic).each do |os, facts|
    context "on #{os}" do
      --- snip ---
    end
  end

This allows you to target just one platform while developing your rspec tests and speed up your test runs considerably. There is a caveat with the above approach, though: it does not work as you'd expect if you try have different checks for different operating systems:

bionic = { supported_os: [{ 'operatingsystem' => 'Ubuntu', 'operatingsystemrelease' => ['18.04'] }] }

focal = { supported_os: [{ 'operatingsystem' => 'Ubuntu', 'operatingsystemrelease' => ['20.04'] }] }

  on_supported_os(bionic).each do |os, facts|
    context "bionic test on #{os}" do
      --- snip ---
      it { is_expected.to contain_file('/tmp/foo') }
    end
  end

  on_supported_os(focal).each do |os, facts|
    context "focal test on #{os}" do
      --- snip ---
      it { is_expected.to contain_file('/tmp/bar') }
    end
  end

While it seems to do the trick, at some point you end up with tests that runon wrong operating systems and may get test failures. And example using the above code:

$ pdk test unit -v
--- snip ---
  bionic test on ubuntu-20.04-x86_64
    is expected to contain File[/tmp/foo]
  bionic test on ubuntu-18.04-x86_64
    is expected to contain File[/tmp/foo]
  focal test on ubuntu-20.04-x86_64
    is expected to contain File[/tmp/bar]
  focal test on ubuntu-18.04-x86_64
    is expected to contain File[/tmp/bar]

If you set /tmp/foo and /tmp/bar to be present only on bionic and focal, respectively, a subset of the tests will fail. So, avoid the above pattern if the desired state varies a lot between different operating systems.

A better approach to handling test variance between operating systems is to always loop through all the supported operating systems, but only run tests for the applicable system. Here's an example from puppet-resolver:

# frozen_string_literal: true

require 'spec_helper'

describe 'resolver' do
  default_params = { 'servers' => ['8.8.8.8', '8.8.4.4'],
                     'domains' => ['example.org', 'example.com'] }

  on_supported_os.each do |os, os_facts|
    context "compiles on #{os}" do
      extra_facts = {}
      extra_facts = { os: { distro: { codename: 'RedHat' } } } if os_facts[:osfamily] == 'RedHat'
      let(:facts) { os_facts.merge(extra_facts) }
      let(:params) { default_params }

      it { is_expected.to compile }
    end
  end

  on_supported_os.each do |os, os_facts|
    context "ubuntu default resolver settings on #{os}" do
      # extra_facts = {}
      # extra_facts = { os: { distro: { codename: 'RedHat' } } } if os_facts[:osfamily] == 'RedHat'
      # let(:facts) { os_facts.merge(extra_facts) }
      let(:facts) { os_facts }
      let(:params) { default_params }

      if os_facts[:operatingsystem] == 'Ubuntu' && ['16.04'].include?(os_facts[:operatingsystemrelease])
        it { is_expected.to contain_class('resolver::dhclient') }
        it { is_expected.to contain_file('/etc/dhcp/dhclient.conf') }
        it { is_expected.to contain_file_line('resolver_config').with('require' => 'File[/etc/dhcp/dhclient.conf]') }
        it {
          is_expected.to contain_exec('restart networking service').with('require' => 'File_line[resolver_config]',
                                                                         'subscribe'   => 'File_line[resolver_config]',
                                                                         'command'     => '/bin/systemctl restart networking',
                                                                         'refreshonly' => true)
        }
      end
    end

    context "ubuntu default resolver settings on #{os}" do
      # extra_facts = {}
      # extra_facts = { os: { distro: { codename: 'RedHat' } } } if os_facts[:osfamily] == 'RedHat'
      # let(:facts) { os_facts.merge(extra_facts) }
      let(:facts) { os_facts }
      let(:params) { default_params }

      if os_facts[:operatingsystem] == 'Ubuntu' && ['18.04', '20.04', '22.04'].include?(os_facts[:operatingsystemrelease])
        it { is_expected.to contain_class('resolver::systemd_resolved') }
        it { is_expected.to contain_file('/etc/systemd/resolved.conf.d') }
        it {
          is_expected.to contain_file('/etc/systemd/resolved.conf.d/50_puppet_resolver.conf').with('notify' => 'Exec[restart-systemd-resolved]',
                                                                                                      'require' => 'File[/etc/systemd/resolved.conf.d]')
        }
        it {
          is_expected.to contain_exec('restart-systemd-resolved').with('command' => 'systemctl restart systemd-resolved',
                                                                          'refreshonly' => true)
        }
      end
    end

    context "redhat default resolver settings on #{os}" do
      extra_facts = {}
      extra_facts = { os: { distro: { codename: 'RedHat' } } } if os_facts[:osfamily] == 'RedHat'
      let(:facts) { os_facts.merge(extra_facts) }
      let(:params) { default_params }

      if os_facts[:operatingsystem] =~ %r{RedHat|CentOS|Rocky} && ['7', '8'].include?(os_facts[:operatingsystemmajrelease])
        it { is_expected.to contain_class('resolver::dhclient') }
        it {
          is_expected.to contain_file('/etc/NetworkManager/conf.d/dns-dhclient.conf').with('ensure' => 'file',
                                                                                              'notify' => 'Exec[restart networking service]')
        }

        it {
          is_expected.to contain_exec('restart networking service').with('require' => 'File_line[resolver_config]',
                                                                       'subscribe'   => 'File_line[resolver_config]',
                                                                       'command'     => '/bin/systemctl restart NetworkManager',
                                                                       'refreshonly' => true)
        }
      end
    end
  end
end

It seems that "pdk test unit" does directly not support running a subset of tests using rspec tags, which is a shame.

More on Puppet testing:

Want to talk to an expert?

If you want to reach us, just send us a message or book a free call!
Categories

Tags

#aad #Access #acl #alertmanager #ansible #ansible module development #Apache #API #augeas #authentication #authorization #automation #automatization #aws #azure #backup #bash #bitbucket #buildbot #cache #centos #cloud #cloud-init #cloudflare #cloudfront #cluster #connectionsJpa #control repo #custom fact #database #debian #devops #digital sovereignty #DNS #docker #domain mode #duplo #ejabberd #email #encryption #erb #europe #eyaml #fabric #facter #facts #fargate #fedora #file #finnish #foreman #freeipa #git #github #gitlab #gnome #google #grafana #hammer #hiera #IAM #import #infinispan #Infrastructure as Code #ipmi #irc #jboss #jdk #jenkins #JMESPath #kanban #keycloak #librarian-puppet #librenms #linkedin #Linux #Location #loop #marketing #mautic #Mellon #mfa #monitoring #mysql #nagios #network-manager #oauth #oauth2 #office365 #open source #openvpn #oxygen #packer #paranormal #pdk #people #php #pkcs7 #pomodoro #Powershell #preseed #presentation #profiles #prometheus #provisioning #puppet #puppet-bolt #puppet-litmus #puppetboard #puppetdb #Puppetfile #puppetserver #puppet types and providers #pxeboot #qemu #quality #r10k #recruitment #redirect #Restrict #Reverse Proxy #roles #rspec #ruby #SAML #sem #shell #showsql #snmp #snmpd #software developement #spam #ssh #sso #standardization #systemd #systemd-resolved #teams #terraform #ubuntu #user-data #vagrant #vanity awards #variable #vim #virtualbox #visualstudio #webdevelopment #wildfly #Windows #wireguard #wordpress #workflow #x11 #xmpp #zimbra
We are
 Puppeteers
menucross-circle