We use Vagrant and Virtualbox a lot because it is an excellent tool for infrastructure code development. Unfortunately it is also very prone to breakage during operating system upgrades, especially if you depend on many plugins. When I upgraded from Fedora 32 to 34 Vagrant broken even more than usual due to conflicting rspec dependencies. Basically every Vagrant plugin refused to install with this error message:

$ vagrant plugin install vagrant-vbguest
Installing the 'vagrant-vbguest' plugin. This can take a few minutes...

Vagrant failed to properly resolve required dependencies. These
errors can commonly be caused by misconfigured plugin installations
or transient network issues. The reported error is:

conflicting dependencies rspec (= 3.10.0) and rspec (= 2.14.1)
  Activated rspec-2.14.1
  which does not match conflicting dependency (= 3.10.0)

  Conflicting dependency chains:
    rspec (= 2.14.1), 2.14.1 activated

  versus:
    rspec (= 3.10.0)

  Gems matching rspec (= 3.10.0):
    rspec-3.10.0

The rspec dependency conflict does not sound particularly scary for somebody who is not developing Vagrant or Vagrant plugins. Therefore ignoring the dependencies is likely to be just fine:

Some of our customers use Microsoft’s Office 365. Due to that they also use Microsoft Teams for collaboration. Recently, out of the blue, I ran into a “redirect loop mystery” in Teams. Basically I was able to login to our Teams (which we only use when recording video calls). But when I tried to switch to a customer workspace I was able to get in, but got immediately redirected back to our own workspace. This issue affected both the official Teams client for Linux, as well as the Chrome browser. Fortunately this was a well-known issue that could be fixed by relaxing cookie security settings in the browser.

Sometimes Terraform stalls when trying to remove AWS EC2 security groups and Terraform does no give any hint as to what is wrong. The problem is caused by that security group being attached to an EC2 instance or network interface.

Interestingly Terraform messes up the order of the AWS API calls even when it (attempts to) remove the security group and detach it from all EC2 instances that use it. In other words, Terraform tries to remove the security group before it has been detached from the EC2 instances, which results in the stall:

Update: the problem described in this article seems to be resolved in Fedora 36 (NetworkManager 1.38.0-2.fc36). It is still present in Fedora 35 (NetworkManager 1.32.12-2.fc35).

Update: there a companion article about solving this problem for Wireguard in here.

In an the Manage Linux client DNS settings in OpenVPN and Wireguard article we described how to use the update-systemd-resolved script to update systemd-resolved per-link DNS servers automatically based on DNS servers pushed from an OpenVPN server. This is very convenient if you have to be connected to multiple remote locations each with their own DNS servers and zones.

We maintain a rather complex AWS environment built with Terraform that uses nested modules and lots of per-resource provider configurations. Recently the pain of staying with Terraform 0.11 became too much, so we had to start the migration to Terraform 0.12, with 0.13 going to follow soon after. In general the official upgrade instructions were pretty good. Of course there was a lot of manual fixing that had to be done, and as I did not keep detailed notes I can’t recall all the steps I went through. But I think all or most of those can be found from other sources.

Occasionally when working with feature branch and updating types and providers you may run into environment leakage issues. They manifest themselves as issues that make no particular sense and are caused by the wrong version of a type/provider such as puppetlabs-apt (see this issue) leaking into your feature environment. This can happen when you update external modules in your Puppetfile. The solution (or band-aid) is fortunately fairly easy. On the Puppetmaster run this command:

I’ve stumbled upon interesting access denied problems with S3-based Terraform state files recently. Suppose you have two or more Terraform root modules which use the same bucket for storing the state and just use a different key (=state file):

terraform {   backend "s3" {     bucket = "terraform-state"     key    = "root-module-1"     region = "eu-central-1"   } }

and

terraform {   backend "s3" {     bucket = "terraform-state"     key    = "root-module-2"     region = "eu-central-1"   } }

However, trying to access the state file in one root module works, but on the other you get a generic S3 Access Denied error: