Some of our customers use Microsoft’s Office 365. Due to that they also use Microsoft Teams for collaboration. Recently, out of the blue, I ran into a “redirect loop mystery” in Teams. Basically I was able to login to our Teams (which we only use when recording video calls). But when I tried to switch to a customer workspace I was able to get in, but got immediately redirected back to our own workspace. This issue affected both the official Teams client for Linux, as well as the Chrome browser. Fortunately this was a well-known issue that could be fixed by relaxing cookie security settings in the browser.

A few days ago I looked at our Azure costs and they were around $8. That’s a ridiculously small amount, except for the fact that we don’t really use Azure for anything. So I started investigating.

After having published the Configuration Manager (Puppet master installer image) and LibreNMS in Azure Marketplace I knew that we had a bunch of BLOBs lying around. I knew how to find these blobs using the Storage Explorer. So I tested wiping them all out, as I knew Azure does not delete them immediately, but only after one day. Then I attempted to launch Configuration Manager and LibreNMS from the Azure Marketplace GUI. My thought was that those blobs had already been copied elsewhere (for global distribution) and would not be used by the Marketplace anymore, but it looked like I was wrong. At least I got odd behavior when trying to launch those images and reverting the blob deletion fixed things.

Update: the problem described in this article seems to be resolved in Fedora 36 (NetworkManager 1.38.0-2.fc36). It is still present in Fedora 35 (NetworkManager 1.32.12-2.fc35).

The title of this blog post may seem familiar to some, as I solved this problem for OpenVPN earlier. I won’t repeat myself here, so please check the OpenVPN article to understand why this is happening.

Wireguard servers - unlike OpenVPN servers - are not able to push any settings such as DNS servers to their clients. However, Wireguard client itself is able to run PostUp commands to configure per-link DNS. The same approach can be used with OpenVPN “up” scripts as well, if you prefer having full control at the client-side. Here’s a Wireguard example:

Puppeteers has launched a new service product which allows you to gain control of your Cloud resources by having them imported to Terraform. This allows you to manage changes to your Cloud and radically reduces the need for the pesky “who changes this and why?”, “who set this up and is it still needed?” and “what does it do?” discussions. As a bonus you get much better visibility into your cloud.

Puppet resources like Package, File or Service are idempotent. In other words, you define the state of the resource and Puppet enforces it. The resources can be used in Bolt plans as well, which frees you from having to do state checks yourself. However, if the thing you’re managing is not a Puppet resource you have to check its state manually and use that information to drive login in your plan. Here’s a crude but functional PoC for checking if a node’s certificate is present on the Puppetmaster and acting accordingly:

When working with Puppet and Puppet modules in particular you quickly notice that there are several ways to manage module dependencies. Today I learned about a new way introduced in Puppet Bolt 3.x, so I thought I’d compile a list that celebrates this diversity:

• metadata.json: used primarily to dependency metadata to Puppet Forge. Some tools like librarian-puppet and puppet-litmus use / can use this file to download dependency modules. The downside is that you can’t define the location of the modules, so you’re limited to Forge modules.

Sometimes Terraform stalls when trying to remove AWS EC2 security groups and Terraform does no give any hint as to what is wrong. The problem is caused by that security group being attached to an EC2 instance or network interface.

Interestingly Terraform messes up the order of the AWS API calls even when it (attempts to) remove the security group and detach it from all EC2 instances that use it. In other words, Terraform tries to remove the security group before it has been detached from the EC2 instances, which results in the stall:

Here’s a recording on our webinar on March 9, 2021:

• Infrastructure as Code

We covered the very basics of infrastructure as code, version control, quality assurance techniques and tools such as Puppet, Terraform, Ansible and Puppet Bolt.

The webinar slides:

Puppeteers-webinar-Infrastructure-as-Code-presentation-210309Download

Puppet Litmus is a Puppet acceptance test harness that leverages on many existing, proven technologies. In the long run Litmus aims to replace Beaker. However, the latter is currently still the only way to run multi-machine (e.g. cluster) acceptance tests in a standardized way. For example we use Beaker for our Keycloak domain mode cluster tests in puppet-module-keycloak. On a high-level the test process is the same as with Beaker:

• Provision the SUT (“System Under Test”)

Puppet Litmus allows creating SUTs (“System Under Test”) with Vagrant using Virtualbox or Hyper-V as the virtualization backend. However, the default virtual machine settings like 1024MB of memory are only suitable for very simple modules and applications. For example, Litmus testing our puppetmaster-installer will fail with the default settings.

Litmus documentation tells us that you can define parameters in provision.yaml which get then converted to environment variables. What it does not tell what to use these environment variables for. Looking at the Vagrant code in Litmus you can see that those environment variables can be used to drive the Virtualbox settings, among other things: