Hints-and-Tips on Puppeteers Oy

Getting latest Rocky Linux AMI ID

Mon, 02 Dec 2024 00:00:00 +0000

Introduction

Rocky Linux 9 is a bug-by-bug clone of Red Hat Enterprise Linux 9 (RHEL 9). We generally recommend using RHEL 9 for mission-critical workloads, but for less important services Rocky Linux 9 is a very good choice if you’re in the Red Hat land already. Official Rocky images are only available in AWS Marketplace, which complicates things when you need to programmatically determine the latest version. This is a common requirement when you do infrastructure as code, for example with Ansible, Terraform or Packer. Although we speak of latest Rocky Linux AMI here, everything in here is applicable for any other AWS Marketplace images, for example Alma Linux.

Fixing ansible-playbook hangs caused by SSH timeouts

Fri, 29 Nov 2024 00:00:00 +0000

Introduction

Ansible is an infrastructure as code tool that uses SSH as its transport mechanism. When the targeted nodes are close, latency-vise to the Ansible controller things usually work fine with the default settings. However, when the targets are far away or latency is big, you may notice that ansible-playbook hangs on some long-running task and never recovers. When you rerun the playbook it may proceed further and then hang at some other task. Or the same task. This is incredibly frustrating, but fortunately also fairly easy to fix.

Alldomains.hosting IPv6 support

Wed, 25 Sep 2024 00:00:00 +0000

We’ve recently started using alldomains.hosting for what we call “programmatic emails”. For that purpose alldomains.hosting is dirt cheap (~1€/year/email address), which makes one quite motivated to suffer through the confusing website to figure out how to buy stuff from them. Once you’ve managed to buy your domain and/or mail services they just “tend to work”. Alldomains.hosting IPv6 support, or lack of it, may be a problem for you, though.

For example, in AWS EC2 you nowadays really want to avoid assigning public IPv4 addresses to EC2 instances as they cost quite a lot. So, you set up an IPv6-only egress-only internet gateway in your VPC and, for the most part, everything “just works”. Then you try to add mail delivery via alldomains.hosting and IPv6 and “boom”, it does not work. According to their support (as of 25.9.2024) their mail servers lack IPv6 support.

Stop EC2 instances automatically with Terraform

Thu, 04 Jan 2024 00:00:00 +0000

Introduction

AWS EC2 is a very convenient place for experimenting with and testing things on virtual machines (or “EC2 instances”). However, it is also very easy to forget such EC2 instances running for no reason, which can become very costly very quickly. Fortunately In AWS computing (e.g. CPU, memory and I/O usage) and storage (disk) usage are billed separately. In other words: if your EC2 instance is stopped, you only pay for its storage. This cuts down the costs significantly, especially if your are using powerful EC2 instance types. So, you definitely should to stop EC2 instances automatically, say, every evening, if you don’t need them to be up. There is no built-in way in to do, though, so this article shows a couple ways it can be done.

Excluding down targets in Prometheus

Fri, 11 Aug 2023 00:00:00 +0000

Excluding down targets in Prometheus is easy. It allow you to monitor volatile systems that start and shut down on demand.

Introduction

Prometheus is an open source metrics and alerting solution used to monitor a wide range of things. Unlike many classic network monitoring systems Prometheus is focused on colleting metrics. This makes it quite easy to do things that would be difficult in others systems. For example excluding down targets in Prometheus is quite trivial. This can be highly useful when you have volatile, static targets that come and go - for example Jenkins slaves (or build agents). Even on such targets you’d want to monitor things like systemd service status, memory usage and disk usage.

Keycloak Javascript policy deployment

Wed, 19 Apr 2023 00:00:00 +0000

Introduction

This article tries to include everything you need to know about Keycloak Javascript policy deployment. By Javascript policies I mean authorization policies of Javascript type attached to Keycloak clients that have authorization enabled. If you don’t know what Keycloak authorization services are, you are probably in the wrong place.

While the process of creating and deploying the JAR files is pretty well documented, it is not altogether clear for the uninitiated how the policies are shown in Keycloak.

Wrong Route53 domain contact details

Thu, 02 Mar 2023 00:00:00 +0000

The symptoms

I recently received and email from Amazon Route53 asking me to verify the contact details for one of our registered DNS domains (replaced here with example.org). And sure enough, I saw wrong Route53 domain contact details in the email:

Subject: Confirm that contact information for example.org is correct.
Message body:

Dear AWS customer,

This message is a reminder to help you keep the contact information for your domain registration up to date. WHOIS records include the following information:

  Domains:
  example.org

  Registrant:
  Name: On behalf of example.org owner
  Organization Name: Identity Protection Service
  Address: PO Box 786
  City: Hayes
  State/Province: Middlesex
  Country: GB
  Postal Code: UB3 9TR

  Administrative Contact:
  Name: On behalf of example.org owner
  Organization Name: Identity Protection Service
  Address: PO Box 786
  City: Hayes
  State/Province: Middlesex
  Country: GB
  Postal Code: UB3 9TR
  Phone: +44.1483307527
  Fax: +44.1483304031
  Email: e793d865-97ab-42ab-805c-50cc36fd9202@identity-protect.org

  Technical Contact:
  Name: On behalf of example.org owner
  Organization Name: Identity Protection Service
  Address: PO Box 786
  City: Hayes
  State/Province: Middlesex
  Country: GB
  Postal Code: UB3 9TR
  Phone: +44.1483307527
  Fax: +44.1483304031
  Email: e793d865-97ab-42ab-805c-50cc36fd9202@identity-protect.org

The contact information provided above is what is shown in the public WHOIS. If your domain is privacy protected, the information will be different than the contact data you have submitted. To verify your contact information, please open the Amazon Route 53 console to view your current contact information and make any necessary corrections:
https://console.aws.amazon.com/route53/home#DomainDetail:example.org

If your information is accurate, you do not need to take any action.

Important
Under the terms of your registration agreement, providing false contact data can be grounds for the cancellation of your domain name registration.

Regards,
Amazon Route 53

These wrong Route53 domain contact details originate from whois data for the registered domain. One might reasonably expect the whois data to match the contact details for AWS Route53 domain. However, in this case the Route 53 contact details were correct, but whois data seemed to be out of sync with those.

Keycloak realm SMTP settings with Ansible

Mon, 27 Feb 2023 00:00:00 +0000

Introduction

Ansible has reasonably good support for managing various aspects of Keycloak. You can use the community.general.keycloak_realm module to handle realm management, including Keycloak realm SMTP server settings. However, in the true Ansible fashion the documentation looks good, but does not help you much. In fact, documentation only mentions that the smtp_server parameter is a dictionary, but that’s it. To be honest, that’s about as helpful as the description of the RealmRepresentation for Keycloak. This article aims to fill this documentation void. I also want to share some general advise for similar cases that pop up when you manage Keycloak programmatically.

Convert a hash into JSON in Puppet

Thu, 22 Dec 2022 00:00:00 +0000

This article shows you how to convert a hash into JSON in Puppet using a simple ERB template that gets its data from Hiera.

Suppose you have this data in Hiera:

myhash:
  mykey:
    foo: one
    bar: two

Converting a hash into a JSON file on the target node is surprisingly easy. First look up the data:

$myhash = lookup('myhash', Hash)

Then create a simple ERB template (here: mymodule/templates/myhash.erb)

Terraform downloads the deprecated azure provider

Tue, 20 Dec 2022 00:00:00 +0000

You are 100% sure that all your Terraform resources are using terraform-provider-azurerm, yet Terraform attempts to download the deprecated “azure” provider:

$ terraform init
Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/local versions matching "2.2.3"...
- Finding latest version of hashicorp/azure...
- Finding hashicorp/azurerm versions matching "3.17.0"...
- Installing hashicorp/local v2.2.3...
- Installed hashicorp/local v2.2.3 (signed by HashiCorp)
- Installing hashicorp/azurerm v3.17.0...
- Installed hashicorp/azurerm v3.17.0 (signed by HashiCorp)
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/azure: provider registry registry.terraform.io does not have a provider named
│ registry.terraform.io/hashicorp/azure
│ 
│ Did you intend to use terraform-providers/azure? If so, you must specify that source address in each module which requires that provider. To see which
│ modules are currently depending on hashicorp/azure, run the following command:
│     terraform providers
╵

You grep the state file and find no references to the “azure” provider. You assume that the cause is some nested module that depends on it, but no, that’s not it. You run “terraform providers” and see that indeed, the “azure” provider is required:

Writing Ansible modules: when and why?

Wed, 23 Nov 2022 00:00:00 +0000

What are Ansible modules?

Ansible modules provide the infrastructure as code building blocks for your Ansible roles, plays and playbooks. Modules manage things such as packages, files and services. The scope of a module is typically quite narrow: it does one thing but attempts to do it well. Writing custom Ansible modules is not particularly difficult. The first step is to solve the problem with raw Python, then you can convert that Python code to an Ansible module

Version locking Ansible Collections

Tue, 01 Nov 2022 00:00:00 +0000

What are Ansible Collections?

Ansible is an infrastructure as code tool used for configuration management, network device management, orchestration and other tasks. Ansible Collections are a way to distribute Ansible content such as roles, playbooks and modules. They can be downloaded from Ansible Galaxy, Git repositories or local directories. Basically collections are a more modern packaging format compared to standalone roles which they are in the process of replacing. Version-locking Ansible Collections allows you to run the exact same Ansible code every time.

Importing DNS records from Cloudflare to Terraform

Fri, 30 Sep 2022 00:00:00 +0000

Computers were supposed to relieve us humans from boring and repetitive jobs. Here we turn this upside down and do the boring and repetitive job of a computer by importing Cloudflare DNS records to Terraform ourselves. Not fun, but someone’s gotta do it sometimes. If you’re reading this, that someone is probably you. Condolences. My hope is that someone lazy enough gets bored enough to write a program to do this. I didn’t. Yet.

Terraform, AWS Cloudfront and CNAMEAlreadyExists error

Sat, 17 Sep 2022 00:00:00 +0000

When you create a distribution, AWS creates several DNS A records with the same name (e.g. d25gma2ea3ckma.cloudfront.net) which point to IPs the distribution is using. Then, typically, you would define CNAME(s) pointing to that cloudfront.net address in your own DNS. Each Cloudfront distribution has a list of aliases, similar to Subject Alternative Names (“SAN”) in SSL certificates. The aliases should match the CNAME(s) you’ve set in DNS.

Now, this is all good until you need to migrate those Cloudfront distributions to, say, a new AWS account. If you attempt to create a new distribution (here with Terraform) with the same alias you will get this error:

Terraform, Azure and MissingSubscriptionRegistration

Sat, 20 Aug 2022 00:00:00 +0000

When deploying with Terraform to Azure you may sometimes encounter errors such as this:

╷                                                                                                                                                             
│ Error: creating Automation Account (Subscription: "83fa140a-2bea-175a-395c-914cae2902ce"                                                                    
│ Resource Group Name: "production"                                             
│ Automation Account Name: "backup-automation"): automationaccount.AutomationAccountClient#CreateOrUpdate: Failure responding to request: StatusCode=409 -- Or
iginal Error: autorest/azure: Service returned an error. Status=409 Code="MissingSubscriptionRegistration" Message="The subscription is not registered to use 
namespace 'Microsoft.Automation'. See https://aka.ms/rps-not-found for how to register subscriptions." Details=[{"code":"MissingSubscriptionRegistration","mes
sage":"The subscription is not registered to use namespace 'Microsoft.Automation'. See https://aka.ms/rps-not-found for how to register subscriptions.","targe
t":"Microsoft.Automation"}]

The problem is that in Azure you may need to register the provider for the service you intend to manage with Terraform. If you add resources from Azure Portal this registration part is handled automation. In the above case the Azure Automation provider was missing.

Finding which MySQL/MariaDB tables are modified by a GUI

Fri, 19 Aug 2022 00:00:00 +0000

I was working with Keycloak realm private/public key automation and it was not immediately obvious where Keycloak stores the keys. Figuring it out was actually easy, and this method applies to any web application that uses MySQL/MariaDB, not just Keycloak.

Anyhow, on Ubuntu, you’d navigate to /var/lib/mysql/name-of-database. For example:

cd /var/lib/mysql/keycloak

Make sure that no changes have been made recently (within one minute):

$ find -mmin -1

Then go to the web application and change whatever you’re trying to locate from the database. Here I changed the Keycloak RSA key priority. Then run the find command again:

AWS: creating AMIs from EBS snapshots shared with you

Mon, 15 Aug 2022 00:00:00 +0000

In AWS EBS (“Elastic Block Storage”) is the underlying technology that (virtual) hard disks of your instances (virtual machines) use. You can take snapshots of those virtual hard disks and use those snapshots to, for example:

Debugging issues with unbootable virtual machines: attach and then mount the snapshot on another virtual machine and investigate what is wrong.
Creating AMI images from which you can launch new instances
Sharing EBS volumes (between regions and AWS accounts), e.g. for purposes of migrating VMs

Here we’ll focus on the last use-case: being able to create copies of virtual machines on another AWS account. The reason why I even bothered writing this blog post is that most of the articles on the Internet do not cover this use-case: they assume you’re working within one AWS account and/or one region. The use-case covered here requires a few extra steps:

Joining nodes to FreeIPA using a non-admin user

Tue, 09 Aug 2022 00:00:00 +0000

Typically Linux nodes are joined to FreeIPA using admin credentials. While this works, it exposes fully privileged credentials unnecessarily, for example when used within a configuration management system (see for example puppet-ipa).

Fortunately joining nodes to FreeIPA is possible with more limited privileges. The first step is to create a new FreeIPA role, e.g. “Enrollment administrator” with three privileges:

DNS Administrators
Host Administrators
Host Enrollment

Then you create a new user, e.g. “enrollment”, and join it to the “Enrollment administrator” role. After that you should be able to join nodes using that “enrollment” user.

Using .sync.yml in Puppet Development Kit (PDK)

Tue, 09 Aug 2022 00:00:00 +0000

Puppet Development Kit is probably the best thing since sliced bread if you work a lot with Puppet. It makes adding basic validation and unit tests trivial with help from rspec-puppet. It also makes it very easy to build module packages for the Puppet Forge.

That said, there is a minor annoyance with it: whenever you run “pdk update” to update PDK to the latest version, all your local changes to files such as .gitignore get destroyed, because PDK creates those files from PDK templates and any local changes are mercilessly wiped. Fortunately there is a way to sync your local changes to the files generated from the templates with .sync.yml. While .sync.yml is is pretty well documented, documentation is lots and lots of words with very few examples. So, here’s a fairly trivial .sync.yml example that ensures local changes to .pdkignore and .gitignore persist across PDK updates:

Use content of a file as a variable with Puppet

Sat, 16 Jul 2022 00:00:00 +0000

Every now and then a need to use the content of a file as a variable on an agent node arises. Here’s one way to do it with the help of a custom fact.

First create a custom fact on the puppet server:

Facter.add(:my_file_content) do
  setcode do
    path = "/path/to/my_file"
    File.read(path) if File.exist?(path)
   end
end

You can confine this to restrict it to be available only on selected nodes, for example:

How to add OpenID to WordPress with Keycloak

Thu, 14 Jul 2022 00:00:00 +0000

Adding OpenID to Wordpress allows for existing users on a domain to connect without having to manage another account. In this setup, we will be using Keycloak to provide the existing accounts which the OpenID plugin will use.

There are a few plugins for Wordpress that allow this functionality, but we will be using OpenID-Connect-Generic by Daggerhart.

Keycloak Client Setup

First step is to create a client in Keycloak. In the Keycloak web UI, create an openid-connect client by going to Clients and click Create.

X11 Connection Rejected

Thu, 28 Apr 2022 00:00:00 +0000

When switching to root after the typical SSH with X11 forwarding enabled this error can appear:

$ X11 connection rejected because of wrong authentication.
$ Error: Can't open display: localhost:10.0

The workaround seems to include copying the MIT-MAGIC-COOKIE-1 from the user who ssh’d in to the root user using xauth. Here’s how:

First verify the $DISPLAY being used and list the MIT-MAGIC-COOKIE-1 used for it :

Enabling system tray on Fedora 35 and 36

Tue, 12 Apr 2022 00:00:00 +0000

System tray is a “legacy” tray where various applications (e.g. Nextcloud, Pidgin and Signal) have an icon with which you can interact with the application without actually opening the main application window. I said “legacy”, because phasing it out was the plan in the Gnome 3 project, but it seems like we’re not getting rid of the tray any time soon, if ever.

In any case the system tray is not enabled by default in recent Fedoras, and the proper way to enable it has changed several times in the recent past as Gnome has been upgraded. This outlines the most recent way to do it, and hopefully this one sticks for longer than a year or so.

Terraform Enum data type

Tue, 12 Apr 2022 00:00:00 +0000

In Terraform you have access to basic data types like bool or string. Defining the data type is a good start for starting to improve the quality of your modules. However, you may want to validate that a certain string matches a list of pre-defined options, and if not, fail validation early.

Terraform, unlike Puppet, does not have a built-in Enum data type. However, you can emulate an Enum like this:

Terraform remote-exec: wait until a webpage is reachable

Fri, 18 Feb 2022 00:00:00 +0000

Terraform’s remote-exec provisioner fails immediately if any command in a script exists with a non-zero exit code. This makes building polling loops a bit more involved than it normally is. So, here is an example loop that checks if a URL can be reached:

while true; do
    curl -k https://puppet:8140/puppet-ca/v1 && break || sleep 3
done

When the URL is unreachable, the “||” will ensure that “sleep” gets run, which returns 0. If the URL is reachable, curl itself will return 0 and “&&” ensures that we break out of the loop.

Showing SQL statements with Keycloak

Fri, 31 Dec 2021 00:00:00 +0000

Sometimes you might find yourself wondering whether there is some paranormal activity going on with your keycloak and its database. To check if things are still in the realm of physical reality, and to restore your child’s faith in the programmer who never makes mistakes, it might be soothing to check what’s actually happening to the database. To do this, create a new property named showSql and give it a value of true in your connectionsJpa spi. This is located in standalone.xml, standalone-ha.xml, or in domain.xml (if you’ve already graduated from kindergarten).

Multiple Bitbucket Cloud accounts with SSH authentication

Tue, 07 Dec 2021 00:00:00 +0000

In Bitbucket usernames are unique across whole of Bitbucket. Moreover, the same SSH key can only be configured for one user. If you registered your Bitbucket account using a corporate email and used your primary SSH key with it, you’re pretty much hosed if you then need to create another corporate Bitbucket account and wanted to use the same SSH key with it - that’s just not going to happen. The solution to this dilemma is to create another SSH keypair and somehow make SSH, Git and Bitbucket co-operate. As the correct solution was not immediately obvious it is documented here.

Managing OpenVPN-based Azure VPN Gateway certificates with easyrsa3

Fri, 19 Nov 2021 00:00:00 +0000

The Azure VPN Gateway supports the OpenVPN protocol (except the “Basic SKU”). Unlike, for example, the commercial Access Server, the VPN Gateway does not have a built-in certificate authority (CA) tool for managing client certificates. And client certificates are essentially a requirement if you need to support clients other than Windows and Mac, such as Linux, iOS or Android. The use of easy-rsa-old as a CA for Azure VPN Gateway is documented quite well in this blog post, but this post is about using the newer easyrsa3 instead. Basic familiarity with easyrsa3 is assumed, so we just go through the steps you’d normally take.

Terraform Azure resource provider registration fails

Tue, 26 Oct 2021 00:00:00 +0000

Terraform has good support for Microsoft Azure through the Terraform Azure provider and the AzureRM backend. However, you may hit a glitch when adding or importing resources if you lack permissions to register Azure resource providers:

│ Error: Error ensuring Resource Providers are registered.
|
│ Terraform automatically attempts to register the Resource Providers it supports
| to ensure it's able to provision resources.
│                                                                                                                                                                                   
│ If you don't have permission to register Resource Providers you may wish to use
| the "skip_provider_registration" flag in the Provider block to disable this
| functionality.
│                                                                                                                                                                                   
│ Please note that if you opt out of Resource Provider Registration and Terraform
| tries to provision a resource from a Resource Provider which is unregistered,
| then the error may appear misleading - for example
|                                                                                                                                                                                   
│ > API version 2019-XX-XX was not found for Microsoft.Foo
|                                                                                                                                                                                   
│ Could indicate either that the Resource Provider "Microsoft.Foo" requires
| registration, but this could also indicate that this Azure Region doesn't support | this API version.
|
│ More information on the "skip_provider_registration" flag can be found here:
|
│ https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs#skip_provider_registration
│
│ Original Error: Cannnot register providers: Microsoft.DataLakeAnalytics, Microsoft.DBforPostgreSQL, Microsoft.AVS, Microsoft.Media, Microsoft.ApiManagement, Microsoft.Search, Mic
rosoft.Maintenance, Microsoft.ServiceBus, Microsoft.HealthcareApis, Microsoft.Relay, Microsoft.DesktopVirtualization, Microsoft.NotificationHubs, Microsoft.SecurityInsights, Micros
oft.DBforMariaDB, Microsoft.Kusto, Microsoft.DataLakeStore, Microsoft.Automation, Microsoft.CognitiveServices, Microsoft.ManagedServices, Microsoft.EventGrid, Microsoft.BotService,

--- snip ---

At first glance this error look cryptic, but it is quite clear once you understand what Azure resource providers are - I like to think of them as “enabled Azure services” or “enabled Azure features”.

Recruiting for a small business - lessons learned, part 2: Tips for job seekers

Fri, 20 Aug 2021 00:00:00 +0000

I previously wrote about takeaways from our first recruitment process as a recruiter in January 2021. This time I’m looking back to the recruitment process in order to gather tips for the other half, job applicants and seekers. Part of the advice are also based on my own experiences as a job seeker and some even on common sense. A recruiter of a bigger company could formulate this article a bit differently but these aspects mattered to us, as a small business.

Making modals responsive in Oxygen Builder for Wordpress

Wed, 14 Jul 2021 00:00:00 +0000

I always thought that it would be impossible for me to make convincing looking websites. But I did learn it and enjoy it very much. But as soon as I’m ready with a new, beautiful web page on my screen, my joy is cut short by the fact that I still need to make it responsive. Learning website development is in fact a double battle: first learning the techniques to make nice things on a web page, and then learning another set of techniques on how to keep the design beautiful on smaller screens. In my experience, the latter battle is way bigger, and thus hope that sharing the lesson I learned will save someone from the agony I had to go through. I’ll focus on modals because making them responsive has been the toughest part for me.

Packer QEMU performance: speed things up with two-stage builds

Thu, 08 Jul 2021 00:00:00 +0000

Introduction

In the “Packer QEMU: Building Ubuntu 20.04/22.04/24.04 VM images” blog post we briefly mentioned ways to improve Packer QEMU performance. In this article we’ll go into the details on how this is implemented. The key in improving Packer QEMU performance is to split the build into two distinct stages:

Operating system installation stage
Provisioning stage

This setup cuts down development time considerably as repeating the provisioning stage becomes a lot faster.

Creating Puppet Bolt groups based on AWS tags

Thu, 01 Jul 2021 00:00:00 +0000

The Using tags in Puppet Bolt aws_inventory target_mapping showed how to use AWS “Name” tag as the target name for Puppet Bolt. Use of tags can be extended to creating Bolt target groups:

groups:
  - name: webservers
    targets:
     - _plugin: aws_inventory
       region:
         _plugin: env_var
         var: AWS_DEFAULT_REGION
       filters:
         - name: tag:Type
           values: [webserver]
       target_mapping:
         name: tags.Name
         uri: public_ip_address

All you need to do is add a “filter” section with one filter. The “name” parameter tells Bolt that the filter is based on tags and what tag to look for. The “Type” that follows “tag” is the name of the tag you want to filter by. The “values” array lists acceptable values for that tag. Any nodes that don’t have the tag (here: “Type”) or where tag value is not acceptable (e.g. “db”) will not be shown in the group.

Using xfreerdp with Vagrant

Tue, 15 Jun 2021 00:00:00 +0000

Vagrant has built-in support for connecting to Windows VMs using the vagrant rdp command. It does basically does the same as vagrant ssh for Linux VMs. There are a few shortcomings, though: vagrant rdp fails to automatically accept the host key of the VM it is connecting to, at least when using it from Linux with xfreerdp. So, you first need to connect to the VM directly once with raw xfreerdp, then vagrant rdp will start working. Then, when you wipe out the VM, you need to repeat the process again. If you have a high resolution screen you will also notice that vagrant rdp will give you a tiny Windows desktop where you need a magnifying glass to read the text.

Installing individual components to Visual Studio 2019 from the command-line

Mon, 07 Jun 2021 00:00:00 +0000

I’ve been doing lots of Windows automation recently. My goal has been to be able to reproducibly create a Windows Server 2019-based Buildbot workers, first in a Vagrant environment, then later in AWS EC2 using pre-built images created with Packer. This task requires installing Visual Studio 2019 build tools automatically and fetching the project’s dependencies with vcpkg automatically. It is possible to install Visual Studio 2019 with Chocolatey quite easily, but what if you need to install additional components to Visual Studio without using the Visual Studio installer GUI? Looking around did not quickly reveal any de facto instructions on how to do it, so this article aims to fill that gap.

Using tags in Puppet Bolt aws_inventory target_mapping

Sat, 05 Jun 2021 00:00:00 +0000

We’re migrating away from Ansible to Puppet Bolt and the fact that Ansible updates broke the old ec2.py inventory script expedited that process. While that inventory script was quite rudimentary, it was able to automatically add human-readable names to the EC2 instance names. So, for example, you could target a node using “server_example_org” if the Name tag was set to “server.example.org”.

Puppet Bolt’s aws_inventory seemed to lack that feature, but after taking a good look at its code this was just a case of missing documentation, which I fixed in this PR. Here’s an example of inventory.yaml for Bolt that maps the value of the “Name” tag to the name of the Bolt target:

Delegating external parties access to AWS

Fri, 14 May 2021 00:00:00 +0000

This blog post shows how to grant access to an AWS account for some external party. For simplicity we will call this external party a “contractor”. If you want to do the same for Microsoft Azure look at this blog post instead.

This method requires the contractor to have its own AWS account, but it has several benefits:

No need to create AWS user accounts for every single person working for the contractor
You can force the contractor’s personnel to use two-factor authentication (2FA)
You can force the contractor to use an additional password (shared secret)
You can disable contractor’s access completely by just removing the IAM role

This approach works equally well for AWS Console access, AWS CLI and tools like Terraform. Some tools may not like the extra security measures required here or be able to assume a role out of the box.

Delegating external parties access to Azure

Fri, 14 May 2021 00:00:00 +0000

In Microsoft Azure you can grant an external user (“contractor”) guest access to your Azure subscription. The approach is quite different from what AWS uses (see Delegating external parties access to AWS).

Here we grant access to john.doe@contoso.com using the Azure Portal. Preferably john.doe@contoso.com would already exist as a Microsoft account.

Login to your Azure Portal
Type “Users” in the search box
Click on the “Users” service
Click on “New guest user”
Select “Invite User”
Fill in the fields. The email address should be the one given by the external user.
Click on “Invite” at the bottom
The external user will receive an invite to your Azure in the email address you defined above.

Now the user has been created. However, it is lacking permissions to do anything useful. We will fix that in the next steps by assigning it a role:

Gitlab SAML to Keycloak Setup

Wed, 12 May 2021 00:00:00 +0000

Introduction

Integrating Keycloak with Gitlab SAML makes it easier to manage users in an organization from a central point. You could manage separate accounts on Gitlab, but for us it makes sense to utilize Keycloak as we are already using it for other applications.

Setting up Keycloak

The first step is to create a Gitlab SAML client in the Keycloak dashboard. With the correct realm selected, navigate to the Clients configuration page and click the create button to add a new client.

Cutting down Azure storage costs for a Marketplace seller

Mon, 19 Apr 2021 00:00:00 +0000

A few days ago I looked at our Azure costs and they were around $8. That’s a ridiculously small amount, except for the fact that we don’t really use Azure for anything. So I started investigating.

After having published the Configuration Manager (Puppet master installer image) and LibreNMS in Azure Marketplace I knew that we had a bunch of BLOBs lying around. I knew how to find these blobs using the Storage Explorer. So I tested wiping them all out, as I knew Azure does not delete them immediately, but only after one day. Then I attempted to launch Configuration Manager and LibreNMS from the Azure Marketplace GUI. My thought was that those blobs had already been copied elsewhere (for global distribution) and would not be used by the Marketplace anymore, but it looked like I was wrong. At least I got odd behavior when trying to launch those images and reverting the blob deletion fixed things.

Making per-link DNS servers added by Wireguard to systemd-resolved survive NetworkManager connection changes

Tue, 13 Apr 2021 00:00:00 +0000

Update: the problem described in this article seems to be resolved in Fedora 36 (NetworkManager 1.38.0-2.fc36). It is still present in Fedora 35 (NetworkManager 1.32.12-2.fc35).

The title of this blog post may seem familiar to some, as I solved this problem for OpenVPN earlier. I won’t repeat myself here, so please check the OpenVPN article to understand why this is happening.

Wireguard servers - unlike OpenVPN servers - are not able to push any settings such as DNS servers to their clients. However, Wireguard client itself is able to run PostUp commands to configure per-link DNS. The same approach can be used with OpenVPN “up” scripts as well, if you prefer having full control at the client-side. Here’s a Wireguard example:

Puppet Bolt: taking action based on exit codes

Tue, 13 Apr 2021 00:00:00 +0000

Puppet resources like Package, File or Service are idempotent. In other words, you define the state of the resource and Puppet enforces it. The resources can be used in Bolt plans as well, which frees you from having to do state checks yourself. However, if the thing you’re managing is not a Puppet resource you have to check its state manually and use that information to drive login in your plan. Here’s a crude but functional PoC for checking if a node’s certificate is present on the Puppetmaster and acting accordingly:

Passing Vagrant parameters to Puppet Litmus

Sun, 07 Mar 2021 00:00:00 +0000

Puppet Litmus allows creating SUTs (“System Under Test”) with Vagrant using Virtualbox or Hyper-V as the virtualization backend. However, the default virtual machine settings like 1024MB of memory are only suitable for very simple modules and applications. For example, Litmus testing our puppetmaster-installer will fail with the default settings.

Litmus documentation tells us that you can define parameters in provision.yaml which get then converted to environment variables. What it does not tell what to use these environment variables for. Looking at the Vagrant code in Litmus you can see that those environment variables can be used to drive the Virtualbox settings, among other things:

Making vagrant-hostmanager UAC prompts go away on Windows

Wed, 03 Mar 2021 00:00:00 +0000

The vagrant-hostmanager plugin is a really nice plugin which helps Vagrant VMs communicate with each other using names instead of IP addresses. Essentially it manages hosts entries for the VMs, acting like a poor-man’s DNS. Now, on *NIX operating systems you typically use sudo, which caches your authorization. So, when hostmanager starts, you type your password once and then it is able to add all the host entries it needs to /etc/hosts. This is not so on Windows: you will get one UAC prompt per modification to c:\windows\system32\drivers\etc\hosts. This will eventually drive you crazy. To fix this you need to permit your Windows user account to modify the hosts-file without privilege elevation. To pull this off, you will need to modify that file’s access control lists. The process is quite convoluted imho, but here it goes:

Fixing vagrant-vbguest for the centos/7 base box

Thu, 25 Feb 2021 00:00:00 +0000

We’re using the centos/7 base box in Vagrant a lot and often hit this problem when bringing up new VMs with Vagrant:

umount: /mnt: not mounted

This error comes from vagrant-vbguest plugin which tried to install the Virtualbox Guest Additions but failed. The error message is misleading, because the real issue is related to upstream (CentOS) yum repositories. As explained in this Vagrant bug report:

“Basically the CentOS that you’re installing is containing an old kernel version which has been removed from CentOS repositories. Vagrant is trying to install Guest Additions, which has as a dependency the kernel-devel package. kernel-devel package must be the same version as the kernel package that you have it installed in your CentOS box, version which is removed from the CentOS repos.”

Creating PXE bootable Virtualbox VMs

Wed, 24 Feb 2021 00:00:00 +0000

We use Vagrant and Virtualbox heavily when developing Puppet code. We also do provisioning of baremetal servers with Foreman. While Vagrant is a really powerful tool, it is not designed for creating VMs that boot from the network (PXE boot). Fortunately Virtualbox itself provides a rich command-line with which you can create and configure virtual machines and configure them to boot from the network. It is assumed below that there is a PXE boot server (e.g. Foreman) created into a hostonly network with Vagrant. Here’s an example script that create a VM with vboxmanage and adds it the same hostonly network interface (vboxnet15) as the PXE boot server:

Terraform resources with dynamic provider values

Thu, 11 Feb 2021 00:00:00 +0000

Sometimes you’d like to pass a dynamic value to a Terraform resource’s provider parameter. This can be done, but some background first.

Terraform allows you to define multiple providers of the same type using provider aliases. This is useful when you’re working with a provider that is tied to a region, the AWS provider being a prime example. The moment you want to create resources in multiple regions you will have to learn provider aliases.

Getting the last inactive vboxnet interface with Ruby

Sat, 30 Jan 2021 00:00:00 +0000

This could possibly be the most esoteric blog post I’ve ever written, but here it goes. So, for reasons even we have difficulty comprehending (hint: related to PXE booting in a Vagrant/Virtualbox environment) we had to find out which inactive vboxnet (hostonly) interface had the highest index. We wanted the code to (hopefully) run on all platforms (Linux, MacOS, Windows) so we wanted to write this code in Ruby. So, this was the brutal but working end-result:

Authenticate and authorize Grafana users from FreeIPA groups via Keycloak

Thu, 28 Jan 2021 00:00:00 +0000

Grafana is a common tool to visualize data from multiple datasources. Perhaps the most common datasource is Prometheus. If an organization has a Single-Sign On solution, it makes sense to authenticate users centrally with that solution That will make authentication easier and friendlier for end users (authenticate once and then access multiple services), and also enable stronger authentication by adding multifactor authentication.

In addition to authenticating user, it is also possible to authorize Grafana users based on their group membership.

Limiting the scope of puppet-rspec tests

Fri, 22 Jan 2021 00:00:00 +0000

Some Puppet modules like puppet-module-keycloak have hundreds of unit tests. That is good for test coverage, but waiting for test results hurts your productivity when you’re developing tests for your new code. There are at least two ways to (temporarily) limit the scope of the tests that you run. First method is baked in into the Puppet Development Kit command-line:

$ pdk test unit --tests=spec/classes/init_spec.rb

This runs only the tests in the specified file.

AWS to Azure resource translation table

Sun, 17 Jan 2021 00:00:00 +0000

We use Terraform for managing our Cloud infrastructure. Our customers typically use AWS and that’s what we’re most familiar with. Each public Cloud has its own terminology, so this page is a translation table between Terraform resource names in the AWS and Azure providers:

AWS	Azure
aws_instance	azurerm_linux_virtual_machine
aws_network_interface	azurerm_network_interface
aws_security_group	azurerm_network_security_group
aws_security_group_rule	azurerm_network_security_rule
aws_vpc	azurerm_virtual_network
aws_subnet	azurerm_subnet
inapplicable	azurerm_resource_group
aws_s3_bucket	azurerm_storage_account
region	location

Mapping of Terraform AWS resource names to Azure resource names

Querying DNS in Puppet code

Wed, 06 Jan 2021 00:00:00 +0000

I recently had to add access control rules to OpenVPN Access Server to permit certain groups to access certain IP addresses. As we manage our Access Server configurations with Puppet using puppet-openvpnas (by the author), we needed a way to parameterize the allowed IP addresses while keeping the Puppet code readable.

This is where puppet-dnsquery came really handy. It allows using data that’s already available in DNS and using it in Puppet code. A simplistic example:

Migrating from Puppet 5 to 6 with Puppet Bolt

Thu, 31 Dec 2020 00:00:00 +0000

The main annoyance with agent-based configuration management is the management of the agents themselves. They usually don’t require any handholding after the initial install, except when you need to migrate them to a new environment. For example, migrate from a Puppet 5-based server to Puppet 6.

The migration process is not overly difficult, but in order to scale to more than a handful of nodes you will need to orchestrate the process. Historically we would have used Ansible for this task, but nowadays we prefer Puppet Bolt, which is more powerful and flexible in our use-cases. The Bolt plans typically end up being ugly, but based on my experience with Ansible Playbooks that tends to be the case anyways. I guess this is the natural result of mixing state-based configuration management, ad hoc commands and orchestration together.

Sed spell to ease moving to Terraform 0.13

Mon, 28 Dec 2020 00:00:00 +0000

When moving from older versions of Terraform 0.12.x to latest 0.12.x (now: 0.12.29) you may notice the following warnings:

Warning: Interpolation-only expressions are deprecated                                                                                                                              

  on backup.tf line 11, in resource "hcloud_server_network" "backup_example_org":                                                                                                 
  11:   server_id = "${module.backup_example_org.id}"

These make sense, but may come as a surprise as earlier 0.12.x versions were perfectly fine with them.

You may have tons of these, so fixing them manually gets tiresome. The fix is not too difficult, but is quite unreadable due to escaping:

Using the Pomodoro technique to control distractions

Sat, 12 Dec 2020 00:00:00 +0000

We - like many others - have the problem that our work gets easily interrupted. Some interruptions come from the inside. For example, you remember “a thing you need to do” and without thinking start doing it immediately. Other interruptions are external, for example a colleague ask you for advise or a customer calls you asking for help. Almost always these things could wait a bit, but the person simply acted on an impulse.

Making Prometheus and Alertmanager work behind a reverse proxy

Wed, 09 Dec 2020 00:00:00 +0000

We maintain a Prometheus and Alertmanager setup where both services are running on the same host and an Apache reverse proxy handles TLS and authentication, authorization and single sign-on with Apache mod_auth_mellon configured as a Keycloak SAML client. This way we can limit access to Prometheus and Alertmanager to people who belong to a certain FreeIPA group.

From end-user perspective Prometheus and Alertmanager are available at these locations:

In this setup both Prometheus and Alertmanager require additional startup options to work correctly. First Prometheus:

Improving Vim's Puppet integration

Thu, 15 Oct 2020 00:00:00 +0000

A recent package upgrade seemed to break Puppet syntax highlighting horribly on my Fedora 32 laptop. Not only did syntax highlighting not work, it was actually completely wrong and confusing. It seemed as if the Puppet file type detection was missing, so vim was probably using some a syntax highlighter for a wrong type. To see which file types are enabled I used :set filetype followed by a couple of tabs (more details on StackExchange).

Restrict access to a location in Apache reverse proxy with TLS

Fri, 04 Sep 2020 00:00:00 +0000

Ever had a case where you needed to use a name based Apache reverse proxy in front of some application server, while restring access to some proxied location at the same time? Here’s how to do it.

First define a virtual host:

<VirtualHost *:443>
    ServerName myserver.example.com

Set the request headers (you are of course using TLS, aren’t you):

RequestHeader set X-Forwarded-Proto "https"
RequestHeader set X-Forwarded-Port "443"

Proxy to some internal address, here to localhost port 8080:

ProxyRequests Off
ProxyPreserveHost On
ProxyPass http://127.0.0.1:8080/
ProxyPassReverse http://127.0.0.1:8080/

Restrict access to the host or networks you need to:

<Location "/my/location/">
 Require ip 10.0.0.0/8
</Location>

Note: this will work with Apache 2.4 and up. With older versions you can use the same idea.

Taming apt timers in Ubuntu 18.04

Wed, 05 Aug 2020 00:00:00 +0000

Ubuntu 18.04 AWS images - and probably other Cloud images as well - have automated upgrades turned on. This is done via two systemd timers, apt-daily.timer and apt-daily-upgrade.timer. The timers kick of periodically as well as when an instance is rebooted. This sounds reasonable until you are provisioning something: you’re unable to install or configure any packages while apt and/or dpkg are running. With a really fresh cloud base image this may not be an issue, but when the base image gets sufficiently old then apt-get upgrade starts taking enough time to cause issues.

Testing connection recovery with ss

Sat, 30 May 2020 00:00:00 +0000

As mentioned in our earlier blog post we created an “email to XMPP bridge” called milter-xmpp for forwarding alert emails from monitoring systems to an XMPP chatroom. It was known that if the connection to the XMPP server ever died the bridge would stop working, but we wanted to know how fast that would happen. It took some days, but eventually the connection died.

The fix ended up being quite trivial, but we needed a way to break the connection. Fortunately the good folks at StackExchange already had a good solution: use the “ss” tool (“another utility to investigate sockets”) which provides output similar to netstat:

Ubuntu and Debian preseeding tips

Mon, 18 May 2020 00:00:00 +0000

Over the years I’ve had to automate installation of Ubuntu and Debian installers a number of times. The primary motivation has always been to avoid all the questions about timezone, locale, partitioning and which packages to install and to not have to babysit the installation. In theory automating the installation is as simple as writing a preseed file. In practice it is much harder for a number of reasons:

Documentation on the Internet around custom installation media and preseeding setup is anecdotal at best - it gets you going and then lets you fall on your face. The official documentation is pretty good, though.
The exact format of the preseed questions varies by operating system version, not just by operating system (Debian, Ubuntu).
In most cases questions about locale and keyboard layout must have answers before the installer preseed is even loaded. This is misleading, because the preseed file does have those questions, even though the answers have zero practical effect for the installer.
There are many ways to accomplish preseeding, most notably by creating a custom installation medium with the preseed file, or by getting the preseed from an URL (“auto mode”).
The default keyboard layout in most Ubuntu/Debian installer is en-US. This is fine, except when you don’t have an American keyboard: then you will be struggling to find characters such as “:” or “/”, which are relatively essential when loading the preseed file from an URL.
It is impossible to remember how to start “automated install” from a vanilla installer menu, let alone figure out what the keyboard and locale settings should look like on the kernel command-line.
The installer is happy to produce unworking configurations (e.g. no bootloader or no kernel) without giving any hints why things are not working.
Each mistake is costly in time, especially if it affects something like Grub configuration which happens at the end of the install.
Because of above, you tend to try to make as many changes to the kernel command line and/or preseeding as possible in one go. This means that when you get things working you don’t know what the proper fix was, and don’t feel like repeating the process to find out.

In order not to be completely negative here is the general recipe for success:

Using snmpwalk to debug SNMPv3 issues

Thu, 14 May 2020 00:00:00 +0000

Our puppet-snmpd module support SNMPv3 very well across a wide range of operating systems. Because o that SNMPv3 is what we use across the board in multiple environments. However, in the rare cases when I have to debug SNMP problems I can never remember the exact snmpwalk command-lines to use. So, here is the usual way to run snmpwalk to figure out if SNMP is providing data:

$ snmpwalk -Os -v 3 -u monitor -l authPriv -a sha -A <auth-pass> -x AES -X <encryption-pass> hostname

Run from the monitoring server (e.g. LibreNMS configured with puppet-librenms) to make sure basic connectivity and access controls are correct as well. Or against localhost if you’re interested in the data and not connectivity.

Manage Linux client DNS settings in OpenVPN and Wireguard

Tue, 31 Mar 2020 00:00:00 +0000

OpenVPN and Wireguard are both used to provide secure, remote access to an organization’s internal services such as network shares (NFS and Samba), desktops and servers (SSH and RDP) or web services. Linux is generally a first-class citizen as far as clients are concerned but with one exception: accessing DNS through the VPN tunnel.

OpenVPN servers have the concept of pushing settings such as DNS servers to client. This happens by emulating a DHCP server. Windows OpenVPN clients for example are able to process this information automatically to update the (per-link) DNS server settings automatically. But Linux traditionally has not had this capability without support from “up” and “down” scripts. When one is using systemd-resolved the traditional ways such as using resolvconf or modifying /etc/resolv.conf directly no longer work or cause issues. Fortunately there is a good script, update-systemd-resolved, for automating the DNS server setup process for systemd-resolved. Basically it takes the list of DNS settings OpenVPN provides and converts them into dbus calls that modify the running systemd-resolved’s configuration.

Giving special treatment to the last entry when iterating over a Puppet hash

Thu, 20 Feb 2020 00:00:00 +0000

When you’re iterating over a hash you may end up having to treat the last hash entry differently. For example, if you’re creating entries to a JSON file you should have “},” in between the entries until you reach the last entry, where you must have just “}” or JSON data will be invalid.

Now, hashes do not have indexes, nor does Puppet support variable reassignment along the lines of $x = $x +1, so you’re out of luck trying to get the length and comparing that to a counter. So, you need some other way to detect when iteration has reached the last entry in the hash.

Debugging rspec-puppet

Fri, 07 Feb 2020 00:00:00 +0000

Usually writing rspec-puppet tests is straightforward. Then you run into some problem that does not make any sense. In my case I ran into a couple of duplicate resource declaration issues when adding rspec tests to our puppetmaster-installer. The error message was not that useful:

$ pdk test unit --tests=spec/classes/puppetboard_spec.rb
--- snip ---
 Failures:
 1) puppetmaster::puppetboard minimal settings on ubuntu-18.04-x86_64 is expected
    to compile into a catalogue without dependency cycles
    Failure/Error: it { is_expected.to compile }
      error during compilation: Evaluation Error: Error while evaluating a
      Resource Statement, Duplicate declaration: Class[Apache] is already
      declared; cannot redeclare (file: /home/samuli/opt/puppeteers/puppet-
      puppetmaster/spec/fixtures/modules/puppetmaster/manifests/puppetboard.pp,
      line: 156) (file: /home/samuli/opt/puppeteers/puppet-puppetmaster
      /spec/fixtures/modules/puppetmaster/manifests/puppetboard.pp, line: 156,
      column: 3) on node puppet.example.org
--- snip ---

That particular block of code in puppetboard.pp did include a parameterized call to the apache class. But nothing else (afaics) included that class. Also, in Vagrant the same code worked just fine, which made things seem even more bizarre. To add insult to the injury the error message did not tell where else the apache class was included. That would probably have been way too helpful.

Terraform: Importing resources from a non-default AWS region

Tue, 04 Feb 2020 00:00:00 +0000

I recently - again - had to import resources to Terraform (0.11.14) from AWS that were outside of the default AWS region and were managed in a nested (non-root) module. The root module contained the provider configuration, including multiple aliases to allow creation of resources in different regions, as well as the nested module call, here in a very simple version:

provider "aws" {
  region = "us-west-2"
}

provider "aws" {
  region = "us-east-1"
  alias  = "us-east-1"
}

provider "aws" {
  region = "us-west-2"
  alias  = "us-west-2"
}

module "cloud" {
  param = value
}

The goal was to import some aws_ses_email_identity resources into the “cloud” module. So, originally the cloud module had this:

Modifying XML files with Puppet's Augeas resource

Fri, 31 Jan 2020 00:00:00 +0000

This article is based on an earlier article written in Finnish.

Here I describe how to make changes to an XML file with Augeas - a useful approach when managing the whole file as a template is out of the question. The configuration file used as an example, web.xml, belong to Pwm, and we intend to set the webapp context parameter.

As discussed before it is important to get an idea of how Augeas sees the file before attempting to modify it with Puppet. First we tell which Augeas lens to use (here: Xml) and which file(s) to associate with the Xml lens. This can be done with in one step:

Allowing changes to a Puppet-managed resource outside of Puppet

Wed, 29 Jan 2020 00:00:00 +0000

When a resource is managed by Puppet it is typically managed fully, or not at all. Exceptions to this rule, such as the “replace” parameter in the File resource, are rare. However, sometimes you end up having to change the properties of a resource outside of Puppet without making Puppet overwrite those changes on the next run. A good example is a set of firewall rules managed with puppetlabs-firewall - you generally want to manage the whole ruleset with Puppet so that you can be sure of the state of the ruleset. For example, if you have libvirt installed, a bunch of virbr0-related rules may appear without your consent. On RedHat-based distros you may also see several rules getting removed by puppetlabs-firewall. Or a fellow sysadmin may have added some random iptables rules for debugging purposes. These are good reasons to have

Getting the interface that matches an IP in Puppet

Tue, 14 Jan 2020 00:00:00 +0000

I was recently in a situation where I needed to figure out which interface name matched a particular, static IPv4 address. On a single or a handful of nodes this could have been hardcoded in Hiera, for example, but there were dozens of nodes. The way I solved this was by creating a custom fact that uses the IPv4 address and loops through the interfaces list to find the matching interface name:

Why modules in Puppetfile should be sorted

Mon, 30 Dec 2019 00:00:00 +0000

This is probably one of the shortest blog postings in the history of this blog. But as this topic does not seem to be covered in more generic articles about avoiding Git merge conflicts (e.g. this, this and this), here it goes.

Adding modules to the Puppetfile is a quite common task. When you or your team has multiple feature branches at the same time many of those branches may add new Puppet modules to the Puppetfile. This is fine, unless the new modules are added to the bottom of the Puppetfile: Git will get confused because it sees that multiple commits have changed the exact same place in the Puppetfile.

Testing Puppet feature branches in Foreman

Tue, 26 Nov 2019 00:00:00 +0000

With default settings Foreman does not allow Puppet Agents to specify their own environment when connecting to the Puppetserver. This makes it impossible to use r10k and feature branch workflow like you can with a normal Puppetserver. This behavior can fortunately be reverted easily by modifying Foreman configuration file, /etc/foreman/settings.yaml:

# Allow testing feature branches on nodes
:enc_environment: false

After modifying settings restart the webserver, e.g.

Reprovisioning bare metal Linux servers with Foreman and IPMI

Wed, 25 Sep 2019 00:00:00 +0000

Recently we had to reprovision a bare metal server that was in Foreman and had IPMI support. The goal was to check if our automated “force-join to FreeIPA domain after reprovisioning” logic worked as intented. We did not have access to any graphical IPMI console, nor Foreman’s GUI, so we had to do everything from the command-line. The host we reprovisioned was a CentOS 7 instance. It is assumed that IPMI tools are installed. This post was inspired by the Automate SuperMicro Server Provisioning or Die Trying blog post.

Enabling PHP 7.4 on CentOS 7 with Puppet

Tue, 17 Sep 2019 00:00:00 +0000

The PHP version comes in default CentOS 7 repositories is rather horribly outdated. This is problematic when running Wordpress, which has to be upgraded constantly, and really old PHP versions may not be accepted, security-patched or not. The solution is to get updated PHP from the remi yum repositories. There are two sets of PHP packages as described here:

Install updated packages that replace the old PHP 5 packages (i.e. names are the same)

Lack of default connection settings in Terraform 0.12.x

Thu, 29 Aug 2019 00:00:00 +0000

I was recently bit by a feature (one might call it a bug) in Terraform. We wanted to try out the “Networks” feature in Hetzner Cloud but that required upgrading terraform-provider-hcloud, which in turn required upgrading Terraform to 0.12.x. For the most part migration from 0.11.x code to 0.12.x code was straightforward with some help from the automated migration tool. I even went as far as read some of the migration documentation, but missed one important part: connection blocks no longer have default values. To make the matter worse Terraform 0.12 does not error out if values are unset: instead if just runs the provisioner and then times out. This seems to be related to Terraform bug #21690.

Moving Jenkins user home directory

Thu, 29 Aug 2019 00:00:00 +0000

Jenkins is a continuous integration (CI) / continuous delivery (CD) platform consisting of a Jenkins master - essentially a controller - which launches jobs on Jenkins slaves. Often slaves are controlled via SSH.

Recently I ran into an issue with a legacy Jenkins slave where the “jenkins” where Puppet tried to enforce the home directory of “jenkins” to be /home/jenkins instead of /var/lib/jenkins:

Error: Could not set home on user[jenkins]: Execution of '/usr/sbin/usermod -d /home/jenkins jenkins' returned 8: usermod: user jenkins is currently used by p rocess 802

The usermod command failed because several Jenkins processes (ssh-agent, Java) locked the user from modification. The solution was relatively straightforward:

Foreman and the Puppetserver environment cache

Sat, 17 Aug 2019 00:00:00 +0000

Importing classes is a two-phase process when you’re using r10k with Foreman. Say you’ve created a new class, “role::ipa_server” to the “production” environment; without Foreman it would be sufficient to deploy the change on the Puppetserver with r10k:

$ r10k deploy environment production -vp

At this point Foreman does not, however, know anything about this new class deployed with r10k:

$ hammer puppet-class list --environment production|grep ipa_server

The “Smart proxy” component in Foreman - also known as “foreman-proxy” - is responsible for fetching new classes from Puppetserver. With the latest Puppet versions it uses the Puppetserver API to do the import. It is fairly trivial to update Foreman’s list of classes using the command-line