Over the years I’ve had to automate installation of Ubuntu and Debian installers a number of times. The primary motivation has always been to avoid all the questions about timezone, locale, partitioning and which packages to install and to not have to babysit the installation. In theory automating the installation is as simple as writing a preseed file. In practice it is much harder for a number of reasons:

• Documentation on the Internet around custom installation media and preseeding setup is anecdotal at best - it gets you going and then lets you fall on your face. The official documentation is pretty good, though.
• The exact format of the preseed questions varies by operating system version, not just by operating system (Debian, Ubuntu).
• In most cases questions about locale and keyboard layout must have answers before the installer preseed is even loaded. This is misleading, because the preseed file does have those questions, even though the answers have zero practical effect for the installer.
• There are many ways to accomplish preseeding, most notably by creating a custom installation medium with the preseed file, or by getting the preseed from an URL (“auto mode”).
• The default keyboard layout in most Ubuntu/Debian installer is en-US. This is fine, except when you don’t have an American keyboard: then you will be struggling to find characters such as “:” or “/”, which are relatively essential when loading the preseed file from an URL.
• It is impossible to remember how to start “automated install” from a vanilla installer menu, let alone figure out what the keyboard and locale settings should look like on the kernel command-line.
• The installer is happy to produce unworking configurations (e.g. no bootloader or no kernel) without giving any hints why things are not working.
• Each mistake is costly in time, especially if it affects something like Grub configuration which happens at the end of the install.
• Because of above, you tend to try to make as many changes to the kernel command line and/or preseeding as possible in one go. This means that when you get things working you don’t know what the proper fix was, and don’t feel like repeating the process to find out.

In order not to be completely negative here is the general recipe for success:

Our puppet-snmpd module support SNMPv3 very well across a wide range of operating systems. Because o that SNMPv3 is what we use across the board in multiple environments. However, in the rare cases when I have to debug SNMP problems I can never remember the exact snmpwalk command-lines to use. So, here is the usual way to run snmpwalk to figure out if SNMP is providing data:

$ snmpwalk -Os -v 3 -u monitor -l authPriv -a sha -A <auth-pass> -x AES -X <encryption-pass> hostname

Run from the monitoring server (e.g. LibreNMS configured with puppet-librenms) to make sure basic connectivity and access controls are correct as well. Or against localhost if you’re interested in the data and not connectivity.

OpenVPN and Wireguard are both used to provide secure, remote access to an organization’s internal services such as network shares (NFS and Samba), desktops and servers (SSH and RDP) or web services. Linux is generally a first-class citizen as far as clients are concerned but with one exception: accessing DNS through the VPN tunnel.

OpenVPN servers have the concept of pushing settings such as DNS servers to client. This happens by emulating a DHCP server. Windows OpenVPN clients for example are able to process this information automatically to update the (per-link) DNS server settings automatically. But Linux traditionally has not had this capability without support from “up” and “down” scripts. When one is using systemd-resolved the traditional ways such as using resolvconf or modifying /etc/resolv.conf directly no longer work or cause issues. Fortunately there is a good script, update-systemd-resolved, for automating the DNS server setup process for systemd-resolved. Basically it takes the list of DNS settings OpenVPN provides and converts them into dbus calls that modify the running systemd-resolved’s configuration.

When you’re iterating over a hash you may end up having to treat the last hash entry differently. For example, if you’re creating entries to a JSON file you should have “},” in between the entries until you reach the last entry, where you must have just “}” or JSON data will be invalid.

Now, hashes do not have indexes, nor does Puppet support variable reassignment along the lines of $x = $x +1, so you’re out of luck trying to get the length and comparing that to a counter. So, you need some other way to detect when iteration has reached the last entry in the hash.

Usually writing rspec-puppet tests is straightforward. Then you run into some problem that does not make any sense. In my case I ran into a couple of duplicate resource declaration issues when adding rspec tests to our puppetmaster-installer. The error message was not that useful:

$ pdk test unit --tests=spec/classes/puppetboard_spec.rb
--- snip ---
 Failures:
 1) puppetmaster::puppetboard minimal settings on ubuntu-18.04-x86_64 is expected
    to compile into a catalogue without dependency cycles
    Failure/Error: it { is_expected.to compile }
      error during compilation: Evaluation Error: Error while evaluating a
      Resource Statement, Duplicate declaration: Class[Apache] is already
      declared; cannot redeclare (file: /home/samuli/opt/puppeteers/puppet-
      puppetmaster/spec/fixtures/modules/puppetmaster/manifests/puppetboard.pp,
      line: 156) (file: /home/samuli/opt/puppeteers/puppet-puppetmaster
      /spec/fixtures/modules/puppetmaster/manifests/puppetboard.pp, line: 156,
      column: 3) on node puppet.example.org
--- snip ---

That particular block of code in puppetboard.pp did include a parameterized call to the apache class. But nothing else (afaics) included that class. Also, in Vagrant the same code worked just fine, which made things seem even more bizarre. To add insult to the injury the error message did not tell where else the apache class was included. That would probably have been way too helpful.

I recently - again - had to import resources to Terraform (0.11.14) from AWS that were outside of the default AWS region and were managed in a nested (non-root) module. The root module contained the provider configuration, including multiple aliases to allow creation of resources in different regions, as well as the nested module call, here in a very simple version:

provider "aws" {
  region = "us-west-2"
}

provider "aws" {
  region = "us-east-1"
  alias  = "us-east-1"
}

provider "aws" {
  region = "us-west-2"
  alias  = "us-west-2"
}

module "cloud" {
  param = value
}

The goal was to import some aws_ses_email_identity resources into the “cloud” module. So, originally the cloud module had this:

This article is based on an earlier article written in Finnish.

Here I describe how to make changes to an XML file with Augeas - a useful approach when managing the whole file as a template is out of the question. The configuration file used as an example, web.xml, belong to Pwm, and we intend to set the webapp context parameter.

As discussed before it is important to get an idea of how Augeas sees the file before attempting to modify it with Puppet. First we tell which Augeas lens to use (here: Xml) and which file(s) to associate with the Xml lens. This can be done with in one step:

When a resource is managed by Puppet it is typically managed fully, or not at all. Exceptions to this rule, such as the “replace” parameter in the File resource, are rare. However, sometimes you end up having to change the properties of a resource outside of Puppet without making Puppet overwrite those changes on the next run. A good example is a set of firewall rules managed with puppetlabs-firewall - you generally want to manage the whole ruleset with Puppet so that you can be sure of the state of the ruleset. For example, if you have libvirt installed, a bunch of virbr0-related rules may appear without your consent. On RedHat-based distros you may also see several rules getting removed by puppetlabs-firewall. Or a fellow sysadmin may have added some random iptables rules for debugging purposes. These are good reasons to have

I was recently in a situation where I needed to figure out which interface name matched a particular, static IPv4 address. On a single or a handful of nodes this could have been hardcoded in Hiera, for example, but there were dozens of nodes. The way I solved this was by creating a custom fact that uses the IPv4 address and loops through the interfaces list to find the matching interface name:

This is probably one of the shortest blog postings in the history of this blog. But as this topic does not seem to be covered in more generic articles about avoiding Git merge conflicts (e.g. this, this and this), here it goes.

Adding modules to the Puppetfile is a quite common task. When you or your team has multiple feature branches at the same time many of those branches may add new Puppet modules to the Puppetfile. This is fine, unless the new modules are added to the bottom of the Puppetfile: Git will get confused because it sees that multiple commits have changed the exact same place in the Puppetfile.