We use Terraform for managing our Cloud infrastructure. Our customers typically use AWS and that’s what we’re most familiar with. Each public Cloud has its own terminology, so this page is a translation table between Terraform resource names in the AWS and Azure providers:

Mapping of Terraform AWS resource names to Azure resource names

I recently had to add access control rules to OpenVPN Access Server to permit certain groups to access certain IP addresses. As we manage our Access Server configurations with Puppet using puppet-openvpnas (by the author), we needed a way to parameterize the allowed IP addresses while keeping the Puppet code readable.

This is where puppet-dnsquery came really handy. It allows using data that’s already available in DNS and using it in Puppet code. A simplistic example:

The main annoyance with agent-based configuration management is the management of the agents themselves. They usually don’t require any handholding after the initial install, except when you need to migrate them to a new environment. For example, migrate from a Puppet 5-based server to Puppet 6.

The migration process is not overly difficult, but in order to scale to more than a handful of nodes you will need to orchestrate the process. Historically we would have used Ansible for this task, but nowadays we prefer Puppet Bolt, which is more powerful and flexible in our use-cases. The Bolt plans typically end up being ugly, but based on my experience with Ansible Playbooks that tends to be the case anyways. I guess this is the natural result of mixing state-based configuration management, ad hoc commands and orchestration together.

When moving from older versions of Terraform 0.12.x to latest 0.12.x (now: 0.12.29) you may notice the following warnings:

Warning: Interpolation-only expressions are deprecated                                                                                                                              

  on backup.tf line 11, in resource "hcloud_server_network" "backup_example_org":                                                                                                 
  11:   server_id = "${module.backup_example_org.id}"        

These make sense, but may come as a surprise as earlier 0.12.x versions were perfectly fine with them.

You may have tons of these, so fixing them manually gets tiresome. The fix is not too difficult, but is quite unreadable due to escaping:

We - like many others - have the problem that our work gets easily interrupted. Some interruptions come from the inside. For example, you remember “a thing you need to do” and without thinking start doing it immediately. Other interruptions are external, for example a colleague ask you for advise or a customer calls you asking for help. Almost always these things could wait a bit, but the person simply acted on an impulse.

We maintain a Prometheus and Alertmanager setup where both services are running on the same host and an Apache reverse proxy handles TLS and authentication, authorization and single sign-on with Apache mod_auth_mellon configured as a Keycloak SAML client. This way we can limit access to Prometheus and Alertmanager to people who belong to a certain FreeIPA group.

From end-user perspective Prometheus and Alertmanager are available at these locations:

• https://prometheus.example.org/prometheus

• https://prometheus.example.org/alertmanager

In this setup both Prometheus and Alertmanager require additional startup options to work correctly. First Prometheus:

A recent package upgrade seemed to break Puppet syntax highlighting horribly on my Fedora 32 laptop. Not only did syntax highlighting not work, it was actually completely wrong and confusing. It seemed as if the Puppet file type detection was missing, so vim was probably using some a syntax highlighter for a wrong type. To see which file types are enabled I used :set filetype followed by a couple of tabs (more details on StackExchange).

Ever had a case where you needed to use a name based Apache reverse proxy in front of some application server, while restring access to some proxied location at the same time? Here’s how to do it.

First define a virtual host:

<VirtualHost *:443>
    ServerName myserver.example.com

Set the request headers (you are of course using TLS, aren’t you):

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"

Proxy to some internal address, here to localhost port 8080:

ProxyRequests Off
ProxyPreserveHost On
ProxyPass http://127.0.0.1:8080/
ProxyPassReverse http://127.0.0.1:8080/

Restrict access to the host or networks you need to:

<Location "/my/location/">
 Require ip 10.0.0.0/8
</Location>

Note: this will work with Apache 2.4 and up. With older versions you can use the same idea.

Ubuntu 18.04 AWS images - and probably other Cloud images as well - have automated upgrades turned on. This is done via two systemd timers, apt-daily.timer and apt-daily-upgrade.timer. The timers kick of periodically as well as when an instance is rebooted. This sounds reasonable until you are provisioning something: you’re unable to install or configure any packages while apt and/or dpkg are running. With a really fresh cloud base image this may not be an issue, but when the base image gets sufficiently old then apt-get upgrade starts taking enough time to cause issues.

As mentioned in our earlier blog post we created an “email to XMPP bridge” called milter-xmpp for forwarding alert emails from monitoring systems to an XMPP chatroom. It was known that if the connection to the XMPP server ever died the bridge would stop working, but we wanted to know how fast that would happen. It took some days, but eventually the connection died.

The fix ended up being quite trivial, but we needed a way to break the connection. Fortunately the good folks at StackExchange already had a good solution: use the “ss” tool (“another utility to investigate sockets”) which provides output similar to netstat: