Deep-Dive on Puppeteers Oy

Monitoring Azure with Prometheus

Thu, 31 Oct 2024 00:00:00 +0000

Introduction

Prometheus is a great tool for “cloud-native” metrics gathering and monitoring. It is particularly easy to manage with infrastructure as code methods. Prometheus can collect metrics from various sources, including Azure Monitor. You can use Azure Metrics Exporter to convert Azure Monitor metrics into Prometheus metrics, but doing so is definitely not trivial unless you’re pretty familiar with Prometheus, Azure Monitor and Microsoft Entra ID apps. This article tries to help you make monitoring Azure with Prometheus a breeze instead of a struggle. We won’t go into a lot of detail about Prometheus or Azure Monitor as there’s plenty of information available elsewhere about both of them. Also note that this article is about vanilla Prometheus, not Microsoft’s “Azure Monitor managed service for Prometheus”.

Avoid double taxation when selling to the U.S

Thu, 03 Oct 2024 00:00:00 +0000

Introduction

We sell digital products in AWS Marketplace, Azure Marketplace and Udemy - see our products page for detail. The end customers of our products are located around the world, including the Unites States of America. All the aforementioned platforms a located in the U.S. This means that they comply with the U.S. tax laws both on the state and federal level. Internal Revenue Service (IRS) is responsible for collecting the federal taxes, which are relevant for this article. Foreign (e.g. European) corporations that sell (digital) products or services to U.S. customers need to prove to IRS that it should not withhold federal taxes from the income coming from U.S. end customers. The goal of this article is to help European corporations (companies) avoid double taxation: first by the IRS and then by the tax office of its own home country. We have all the basic information that a typical European corporation (company) needs to do to avoid double taxation when selling services or digital products to U.S. end customers.

Microsoft Entra ID multi-factor authentication in RDP

Fri, 28 Jun 2024 00:00:00 +0000

Introduction

Microsoft Entra ID, formerly known as Azure Activate Directory (AAD), is replacing classic on-premise Active Directory installations slowly but surely. It replaces “legacy” protocols such as LDAP and Kerberos with “modern” protocols such as SAML, OIDC and OAuth2. However, the client-side support in various operating systems such as Windows, Linux and MacOS is spotty, to say the least. You can make things work, but only if the stars (or rather, bits) are aligned just right. This article explains how multi-factor authentication in RDP works with Azure VMs when using Microsoft Entra ID. Note the emphasis on MFA: without MFA you can make logins with Entra ID work more easily by using the normal username + password authentication in RDP.

Using Hiera with Ansible

Fri, 07 Jun 2024 00:00:00 +0000

Introduction

Hiera is a very powerful “key-value configuration data lookup system” used mainly, but not exclusively, with the Puppet configuration management and orchestration system. In Hiera data is stored in in yaml files in, as the name implies, a hierarchy or set of prioritized levels. A typical Hiera lookup goes through the hierarchy starting from the highest-priority level and stops when it finds a match. That match could come from a node yaml file, a deployment (e.g. production or staging) yaml file or from site-wide yaml. What this essentially means is that you can define common, site-wide parameters at the lowest level of the hierarchy and override those when needed at higher levels. How does Hiera and Ansible work together, then? The short answer is “they don’t, really”, but the long answer is more complex than that.

RHEL on Upcloud

Mon, 27 May 2024 00:00:00 +0000

Introduction

Upcloud is a Finnish Cloud provider that focuses on high-quality services. There is no built-in support for installing RHEL on Upcloud, which is quite typical of “small Clouds”. Sometimes you don’t want to use clones (e.g. Rocky Linux) and need real RHEL instead. This is where this article comes handy, as the RHEL installation process in Upcloud is far from trivial. This article focuses on RHEL 9, but the process may work with older and newer versions as well. The process I describe here applies almost 1:1 to any Linux operating system Upcloud does not natively support.

Buildbot build order: how to run certain builds before others?

Tue, 06 Feb 2024 00:00:00 +0000

Introduction

The mechanisms for configuring Buildbot build order is documented fairly well. However, what that documentation lacks is a centralized place where ordering is explained. My personal goal was to run smoke tests first and if those don’t pass then cancel all other build requests. That seemed straightforward, but turned out not to be so. There were many, many ways that could in theory be achieved, but most of those did not work at all, or worked only partially. This article attempts to fill the “how do I order builds in Buildbot” gap, including what can and can’t be achieved with each strategy.

Install RHEL 9 on Hetzner Cloud

Fri, 20 Oct 2023 00:00:00 +0000

Scaling a mountain is about as easy as to install RHEL 9 in Hetzner Cloud in scalable way. At minimum, you need this article and lots of endurance to pull it off. Photo credit.

Introduction

Hetzner Cloud is a really good “small” Cloud with particularly good value for money. While it does not have hundreds of services like AWS or Azure, but it has most of the basic stuff. Hetzner cloud has built-in support for Red Hat Enterprise Linux (RHEL) clones such as Rocky Linux 9. However, you cannot spin up real RHEL 9 instances without spending some time on it. This article will give you all the details you need to install RHEL 9 on Hetzner Cloud. We do that in a way that you can reuse your RHEL 9 installation to spin up multiple servers from one snapshot. This way you don’t have to run the RHEL installer for every RHEL server you want to spin up.

Blackbox Exporter ICMP module and selinux

Thu, 21 Sep 2023 00:00:00 +0000

Blackbox Exporter ICMP module and selinux do not like each other. Same goes for systemd units and ICMP Echo sockets. However, with persistence, skill and a bit of luck you can make it all work together. Photo credit.

Introduction

Blackbox Exporter is an exporter for the Prometheus monitoring system. It is able to probe a target, such as a host or an application from the outside. Blackbox Exporter does probing with modules such as http, tcp, dns, icmp and grpc (details here). The icmp module is special because it does privileged operations that require the CAP_NET_RAW capability and as well as custom selinux policies. This article aims to explain how you can make Blackbox Exporter ICMP module and selinux work together. NOTE: you do not need any of these tricks if you use non-icmp Blackbox exporter modules.

How to reduce RDS storage size in an automated way

Thu, 10 Aug 2023 00:00:00 +0000

Introduction

Amazon RDS is a relational database service available on Amazon Web Services. It is essentially a managed database server with a volume for the data. Both cost money. Having a large dataset is not the only reason have big volume: the volume size determines the number of I/O operations (IOPS) the volume can do. If you have high load on your RDS instance you may very well run out of IOPS which can result in a disaster. Scaling up the volume size is easy to avoid such disasters, but it is much more difficult to reduce RDS storage size. This is because downsizing RDS storage size is not officially supported by AWS. If the load on your RDS instance goes down (e.g. due to refactoring), you may end up having a database with an oversized volume that costs lots of money for no reason.

Downsize AWS RDS Volume

Mon, 08 May 2023 00:00:00 +0000

It is trivial to downsize an AWS RDS instance. It is equally trivial to upsize AWS RDS volumes. However, to downsize AWS RDS volume you need to jump through lots of hoops.

So, why would you ever want to downsize an AWS RDS volume? One reason might be that you increased the volume size simply to get more burst credits for the volume. Then you did refactoring that reduced the need to have that many credits and want to save money. Or maybe you genuinely had a huge amount of data on your RDS instance volume, removed some of it but got stuck with a huge volume.

Keycloak authorization services REST API: paths and payloads

Fri, 24 Mar 2023 00:00:00 +0000

Introduction

The Keycloak Authorization Services allows you to offload your application’s authorization decisions to Keycloak instead of implementing them in your code. This way you can leverage Keycloak’s advanced features like 2FA without any additional development on your part. If you’re unfamiliar with the Authorization Services I suggest having a look at the Keycloak authorization services terminology first. You may also be interested in our sample code for managing Keycloak authorization services programmatically. To be perfectly honest, there is no separate Keycloak authorization services REST API. While the authorization services do support the UMA Protection API, that API is only needed if you want your resource servers (e.g. webapps) to be able manage their own authorization services configuration inside Keycloak. For example, you might want to provide your webapp users the capability to grant access (e.g. read, write) to their own resources to other users. In this article we will not cover the Protection API at all.

Debugging Keycloak OIDC token exchange

Wed, 22 Mar 2023 00:00:00 +0000

Introduction

There’s no substitute for understanding what you’re doing, and that in turn is difficult without seeing what is happening. Debugging Keycloak OIDC problems without understanding what is happening under the hood is no exception to this rule. The purpose of this article is twofold:

Help you learn how Keycloak OIDC token exchange work
Provide you with the tools to debug Keycloak OIDC token exchanges in production environments

For the purposes of this blog post I’ve been using the OpenID Connect Playground application from the book Keycloak - Identity and Access Management for Modern Applications from Packt Publishing. I can recommend that book for anyone who needs to understand Keycloak and in particular the protocols it supports (OAuth2, OIDC and SAML 2.0).

Keycloak authorization services: terminology

Fri, 17 Mar 2023 00:00:00 +0000

Introduction

Keycloak Authorization Services are built on top of well-known standards such as the OAuth2 and User-Managed Access specifications (UMA). You can manage Keycloak Authorization Services programmatically, if needed, as described in our other blog post.

OAuth2 clients (such as front end applications) can obtain access tokens from the server using the token endpoint and use these tokens to access resources protected by a resource server (such as back end services).

In the same way, Keycloak Authorization Services provide extensions to OAuth2 to allow access tokens to be issued based on the processing of all policies associated with the resource(s) or scope(s) being requested. This means that resource servers can enforce access to their protected resources based on the permissions granted by the server and held by an access token. In Keycloak Authorization Services the access token with permissions is called a Requesting Party Token or RPT for short.

Let's Encrypt on AWS: when, where and why

Wed, 15 Mar 2023 00:00:00 +0000

Let’s Encrypt on AWS can be a can of worms if you don’t know what you’re doing. Photo credit.

Introduction

Let’s Encrypt and the ACME protocol enable getting free (as in beer) TLS/SSL certificates. The main use for Let’s Encrypt is to enable HTTPS on web servers. Let’s Encrypt TLS certificates expire pretty quickly, in 3 months, but the idea is that renewal is automated so that you don’t ever really have to worry about expiration, unlike with commercial TLS certificates, which you typically need to manually renew every 12 months. Using Let’s Encrypt on AWS is in some cases a very viable option, but not so much in others. In many cases trying to use Let’s Encrypt certificates leads you down the path of Lovecraftian horrors and madness. Because “it all depends” I’ll try to explain when using Let’s Encrypt is viable in the AWS context and when you really should just give up an use certificates generated by the Amazon Certificate Manager (“ACM”) instead.

Integrating Twilio SMS and Prometheus Alertmanager

Thu, 16 Feb 2023 00:00:00 +0000

Introduction

Prometheus is a widely used, cloud-native open source monitoring solution. Its alerting component is called Alertmanager, which can send alerts to email, Slack and elsewhere. Both Prometheus and Alertmanager fit very well into the infrastructure as code model as well. Twilio SMS is a SMS sending service with consumption-based pricing model with an extensive API. With some effort you can integrate Twilio SMS and Prometheus together as described in this article.

Testing AlertManager webhooks with curl

Mon, 06 Feb 2023 00:00:00 +0000

Introduction

Prometheus is a Cloud-native metrics platforms that is very easy to manage with infrastructure as code tools. Prometheus is often couple with Alertmanager which handles alerting and alert routing. AlertManager has good support for various alert transport (e.g. email or slack) but its alerting capabilities can be extended with custom webhooks. When AlertManager is configured to use a webhook in an alert route, it forwards alerts to an HTTPS/HTTP endpoint instead of handling transporting of the alert itself. AlertManager’s official documentation gives a rough idea of what kind of payload to use and how to configure the webhook in alertmanager.yaml. What is not documented is how you can send test alerts to the webhook without having to trigger real (or fake) alerts in Prometheus. While using fake alerts does work, it is a slow, tedious and invasive process. This article shows one way for testing AlertManager webhooks with curl. In our case the target was to speed up testing of changes to promtotwilio.

Adding new permission with Keycloak authorization services REST API

Sun, 15 Jan 2023 00:00:00 +0000

What does it do, this Keycloak thing?

Dear seasoned keycloaker, as you probably know, Keycloak is a stable, scalable, programmable and otherwise killer platform to centralize all your identity, authentication and authorization needs. Keycloak supports fine-grained authorization policies using Keycloak authorization services. We highly recommend reading our Keycloak authorization services terminology article before this article. if you don’t understand terms, you will have some hard times ahead. The purpose of this article is to give you an example of how to work with the Keycloak authorization services REST API.

Debugging Terraform AWS UnauthorizedOperation errors

Wed, 11 Jan 2023 00:00:00 +0000

Causes for the Terraform AWS UnauthorizedOperation errors

Terraform is an infrastructure as code tool you can use to configure Cloud resources in AWS. When using Terraform AWS provider you frequently run into various UnauthorizedOperation errors when creating, modifying or deleting resources. That happens unless you do what you should not do and let Terraform use an AWS API key tied to an IAM user with full admin privileges. In almost all cases Terraform UnauthorizedOperation errors are caused by missing IAM permissions. It can sometimes be a bit tedious to get to the root cause, because Terraform may not show it in its normal output.

How to disable S3 bucket ACLs

Tue, 10 Jan 2023 00:00:00 +0000

Introduction

AWS recommends that you disable S3 bucket ACLs for all new buckets. To understand why some background information is needed.

AWS S3 providers two ways to manage access to S3 buckets and objects:

Access policies: these are standard IAM policies that you attach to users (“user policies”) or resources (“resource-based policies”). IAM policies are the recommended method for managing access to Amazon S3 nowadays.
Access Control Lists (ACLs): ACLs are an older access control method for AWS S3. ACL map to IAM permissions. ACLs works on the bucket and object (file) file.

Azure Private DNS and VPN Gateway

Tue, 13 Dec 2022 00:00:00 +0000

What is Azure Private DNS?

Azure Private DNS is a DNS service for Azure virtual networks. You can register a private DNS zone to Azure Private DNS and then link that zone with one or more virtual networks. If you enable DNS auto-registration for a virtual network, a new resource (e.g. virtual machines and VPN Gateways) will automatically add its IP address to Azure Private DNS. Resources in the linked virtual networks can use entries in Azure Private DNS to locate each other, instead of relying on IP addresses or the default Azure DNS domain. You can make Azure Private DNS and Azure VPN Gateway work together in several ways, some of which this article will describe.

Cloud automation and infrastructure standardization at Red Hat Open Tour 2022

Sat, 01 Oct 2022 00:00:00 +0000

We participated in Red Hat Open Tour 2022 Tallinn a while back. Johan Wennerberg, who is a Solution Architect for Red Hat Nordics in Stockholm, gave a presentation titled “Gain robust repeatability as self.service, by automating your automation”. Among other things he discussed the importance and use-cases of Cloud infrastructure standardization and automation. Here I willsummarize his views on standardization and automation and explain how they tie in with our focus on quality in the Cloud infrastructure and elsewhere.

Enabling Azure Backup on Linux VMs with Terraform

Mon, 05 Sep 2022 00:00:00 +0000

This article shows you how to enable Azure Backup on Linux VMs. It is recommended to read the Understanding Azure Backup for Linux VMs article first before trying to enable backups with Terraform. Terraform AzureRM provider has three relevant resources:

azurerm_linux_virtual_machine: parameters provision_vm_agent and allow_extension_operations should be true or enabling backups will fail (with or without Terraform)
azurerm_virtual_machine_extension: this can be used to run/enable a VM extension on a VM
azurerm_backup_protected_vm: this resource enables backups on a VM
azurerm_recovery_services_vault: the recovery services vault where backups will be stored
azurerm_backup_policy_vm: backup schedule and retainment rules

I’ll go through the basic process first, which may work for some people. I’ll then take a deep dive into the timeout issues you may or may not encounter in your environment.

Understanding Azure Backup for Linux VMs

Mon, 29 Aug 2022 00:00:00 +0000

Azure Backup is an Azure service that allows, among other things, backing up Windows and Linux VMs in Azure. The backups are essentially virtual machine snapshots, but backing up and/or restoring individual files is also possible. This article tries to explain how Azure Backup and Linux VMs interact and what is required for them to work. If you’re interested in using Terraform to configure Azure Backup, please have a look at the Enabling Azure Backup on Linux VMs with Terraform article.

Terraform: converting an aws_instance network interface into an explicitly managed interface

Tue, 23 Aug 2022 00:00:00 +0000

The aws_instance resource in Terraform can automatically create the default network interface for you. There are cases, however, when you notice that the default network interface is not enough anymore, and modifying it via the limited aws_instance parameters is not sufficient. In these cases you can convert the interface into an aws_network_interface resource, but the process is far from straightforward.

The first step is to create a Terraform resource for the network interface:

Managing custom realm keys in Keycloak programmatically

Fri, 19 Aug 2022 00:00:00 +0000

What are Keycloak realm keys?

Keycloak’s authentication protocols make use of private and public keys for signing and encrypting, as described in the official documentation. These keys are realm-specific, and by default managed internally in Keycloak. So, when you create a realm using the Keycloak Admin API, kcadm.sh or manually using the Web UI, new keypair(s) get generated automatically. These are called “Managed keys”. However, you can also use custom realm keys in Keycloak, which is what this article is about. Using custom realm keys allow you, for example, to use a single keypair in multiple realms. When you import a custom private key, Keycloak uses it to automatically create a certificate. If you want, you can pass both a private key and a certificate signed with that private key instead. That way you can bypass Keycloak’s automatic certificate generation, should you need to.

Modern cronjob part 1: Azure Automation with Terraform

Thu, 11 Aug 2022 00:00:00 +0000

Microsoft Azure has a nice service for scheduling tasks called Azure Automation. While Azure Automation is able to other things as well, such as being able to act as a Powershell DSC pull server, we’ll focus on the runbooks and scheduling. Runbooks are scripts that do things, e.g. run maintenance and reporting tasks. Runbooks often, but not necessarily manipulate objects in Azure. Runbooks are serverless like Azure Functions and the pricing is therefore similar - you pay for the amount of computing resources you use and don’t have to pay for a server that’s mostly idle.

Multi-part cloud-init provisioning with Terraform

Thu, 28 Apr 2022 00:00:00 +0000

Cloud-Init is “a standard for customizing” cloud instances, typically on their first boot. It is allows mixing state-based configuration management with imperative provisioning commands (details in our IaC article). By using cloud-init most of the annoyances of SSH-based provisioning can be avoided:

Having to use (possibly shared) SSH keys for provisioning
Having to have direct network access to the VM being provisioned (may require a VPN connection)

That said, neither cloud-init itself, nor its use within Terraform is particularly well documented. Therefore it can be an effort to create cloud-init-based provisioning that works and adapts easily to different use-cases. This article attempts to fill that gap to some extent.

Data-driven Terraform: Kubernetes cluster in Hetzner Cloud

Fri, 15 Apr 2022 00:00:00 +0000

Terraform does not have a particularly strong decoupling between data and code, at least not from a best practices perspective. It is possible and useful, however, to use data to define Terraform resources - if not for any other reason but to reduce code repetition for common resources that require defining lots of parameters. Here’s an example of setting up a Kubernetes cluster in Hetzner Cloud using Terraform with a “data-driven” approach:

Debugging Puppet Bolt inventory plugins

Mon, 14 Mar 2022 00:00:00 +0000

Puppet Bolt handles inventories in a very flexible and powerful manner: you can combine static target definitions and different targets into a single inventory. For example, you can have an inventory which defines some static node names combined with the AWS inventory, or one that combines static nodes with the Vagrant inventory.

Puppet Bolt inventory plugins do their magic and then output what is essentially a static inventory generated dynamically. When dynamic inventory plugins do not work (as in my case) it can be challenging to figure out what is wrong, let alone how to fix it. However, there’s is a very handy way to print out a static version of the entire inventory, including the dynamically created content:

Are Corporate Vision awards fake?

Thu, 10 Feb 2022 00:00:00 +0000

This was a burning question in my head in September 2021 but didn’t have much luck googling it. Here’s our experience and analysis of Small Business Awards by Corporate Vision. Read on to find out whether the awards handed out by Corporate Vision awards truly are a legit or not.

How did we hear about Corporate Vision?

Corporate Vision contacted us in September 2021 announcing us that we were nominated as candidate for being the best company in the ‘IT Infrastructure Management Specialists - Finland’ category for Small Business Awards. At that time (before this blog post) there was very little information online about Small Business Awards or the company behind them. What I learned at that time was that they seemed to be a so-called ‘vanity award’, where winners have been preselected and everything else was just for show. We could not be absolutely sure about that then, and we thought that “winning” might have some marketing value, so we decided to play along and see where it leads us. I will first go through the process, then I’ll try to give an answer the question in the title.

Mautic spam prevention

Mon, 07 Feb 2022 00:00:00 +0000

Mautic is a widely used open source email marketing automation application written in PHP. Email marketing is typically used in conjunction with inbound marketing done by asking people to give their email address in exchange for something, like a free ebook or a newsletter with good content. As you’re asking for the email address on a publicly facing webpage, spambots will find your form, and start misusing it to send your “Welcome” emails all over the place. This in turn may result in lots of bounces and/or complaints, which may get you locked out from your email service provider unless you pay attention. So, you need a way to block or fool those spam bots to prevent such issues, and to keep the Mautic database clean.

Conditional provisioner blocks in Terraform

Wed, 05 Jan 2022 00:00:00 +0000

I’ll start with a spoiler: what the title suggests is not possible. It is, however, possible to accomplish this with cloud-init and Terraform templates as described in the Multi-part cloud-init provisioning with Terraform blog post.

If you need to use SSH/WinRM provisioning, then there are various workarounds you can apply, and this article explains some of them and goes into depth into one in particular.

The first workaround is relatively clean, but is not usable in all cases, especially when working with complex modules. The idea is to have two resources, here of type aws_instance. Those resources are identical except for the provisioner blocks:

Dealing with multiple AWS accounts with one Keycloak client for Single-Sign On

Thu, 18 Nov 2021 00:00:00 +0000

This article assumes that the user backend for Keycloak is FreeIPA. Regardless of that the instructions will apply to any other setup with minor modifications. Here we use two different AWS accounts renamed to 123412341234 and 567856785678 to protect the personal information of the innocent. The Keycloak staging cluster on which this integration was done is called id-staging.example.org and the production cluster is called id.example.org. The Keycloak realm is EXAMPLE.ORG. Replace those with the values correct for your environment.

Allowing external email forwarding in Office 365

Tue, 16 Nov 2021 00:00:00 +0000

We use Zimbra as our main email server. We also have Office 365 subscription to make working with our clients a bit easier. The challenge is that when customers send us, say, Teams meeting invites, they typically use autofill and the email gets sent to our Office 365 mailboxes which nobody really looks at.

It is possible to fix this by configuring email forwarding. That is theoretically possible out of the box on a per-user basis in https://outlook.office.com/mail/options/mail/forwarding. However, when you actually try it out you will get an error message in your Office 365 inbox because forwarding is disabled on the organizational level by default:

Enabling AWS EC2 instance automatic recovery with Terraform

Thu, 28 Oct 2021 00:00:00 +0000

AWS EC2 instances are subject to two types of status checks (AWS docs):

System status check (issues with the underlying hardware/networking: “the AWS side”)
Instance status check (issues with the OS, e.g. OOM, file system corruption, broken networking, etc: “our side”)

The official AWS EC2 instance recovery documentation claims that automatically recovering from an EC2 instance failure is only possible for system status checks, but the AWS documentation is outdated: Cloudwatch metrics are available for both types of check failures and people have set up recovery for both types of failures using AWS CLI.

Extending snmpd with custom scripts

Fri, 16 Jul 2021 00:00:00 +0000

Traditional network monitoring systems tend to get their monitoring data via SNMP. In case of Linux snmpd is usually the system application that’s responsible for providing that data. The data consists mostly, but not entirely, of metrics. For example, the data contains strings such as operating system version, administrative contacts and network interface names.

It is possible to make snmpd run custom scripts and programs to gather more data from the system. The options are described in snmpd.conf man page, but here are the most important ones for this use-case:

Packer QEMU: Building Ubuntu 20.04/22.04/24.04 VM images

Mon, 05 Jul 2021 00:00:00 +0000

Introduction

Packer QEMU builder allows you to build virtual machine images from scratch. You need a Packerfile, an automatic installation configuration file (autoinstall, kickstart or preseed) and the operating system installer (e.g. ISO). The benefit of Packer QEMU in particular are that you can have fine-grained control over the virtual machine image content such as disk partitioning. Moreover you don’t need to rely on official base images produced by other people, such as the official Ubuntu Cloud images.

Making wpcal.io work with Zimbra calendars

Thu, 01 Jul 2021 00:00:00 +0000

Scheduling meetings with email is annoying. When everyone is on the same timezone that kind of works, but tends to be slow. When you factor in timezone differences the process becomes really error-prone and slows down even more, often to several days. For this reason we wanted to provide people - including our customers - a way to easily schedule meetings with us.

We’re special in the sense that we self-host almost everything we use. For email and calendars we use Zimbra and for our website we use Wordpress. Most people seem to use Calendly as a meeting scheduler, but we ended up choosing wpcal.io which is a native Wordpress plugin and seems to work quite nicely.

Testing Puppet feature environments with Puppet Bolt

Fri, 18 Jun 2021 00:00:00 +0000

Puppet feature environments are an excellent way to test code before deploying it, typically to production. They allow testing Puppet runs on no-operation mode across the whole node population managed by Puppet. There are sometimes cases where your code changes potentially impact many nodes and you’re not exactly sure of their scope or effect. In that case it helps to test your code on all Puppet-managed nodes. The challenge is that Puppet open source does not have any built-in mechanism to do that. This is where we can make good use of Puppet Bolt.

Puppet types and providers development part 5: self.instances

Sun, 30 May 2021 00:00:00 +0000

This blog post is a part of this blog post series:

The self.instances is a provider class method that is used to produce an array containing all resources found from the system. For example, a yum package provider might run “rpm -qa” in self.instances to get a list of packages installed on the system. This serves two main purposes:

Allows using of “puppet resource type” to produce Puppet code for all resources of type found on the system.
Data created by self.instances can be passed to self.prefetch to cache resource properties. The cached properties are then reused for every resource of that provider type. For example, a yum package provider might create a list of packages installed on a system when Puppet encounters the first “yum” package resource of “yum” type. That cache would then be reused, thus removing the need to individually check each package’s state.

As discussed in Puppet types and providers development part 4: caching resource properties to improve performance self.instances or self.prefetch are not really suitable for scenarios where you need to manage (multiple) remote systems using the same provider, because things like connection would be given in the Puppet resource definition are at the provider instance level, to which provider class level methods like self.instance and self.prefetch do not have access to.

Puppet types and providers development part 6: mysteries of self.prefetch

Sun, 30 May 2021 00:00:00 +0000

This blog post is a part of this blog post series:

I will open this blog post with a quote from the famous Gary Larizza:

After wading the waters of self.prefetch, I’m PRETTY SURE its implementation might have come to uncle Luke after a long night in Reed’s chem lab where he might have accidently synthesized mescaline.

http://garylarizza.com/blog/2013/12/15/seriously-what-is-this-provider-doing/

After reading Gary’s rather elaborate explanation about self.prefetch I still did not fully get what self.prefetch actually does and the official documentation surely does not help:

Limiting concurrent builds in Buildbot

Fri, 21 May 2021 00:00:00 +0000

Buildbot is a continuous integration framework which many open source projects seem to be using. Unlike continuous integration applications or automation servers like Jenkins Buildbot does not make many assumptions about your use-case. In fact, when you build your CI or CD pipeline you’re actually writing Python code and hence have all the flexibility of Python directly at your disposal. In other words, you’re not being blocked or obstructed by assumptions hidden in the tool itself, like often tends to happen with Jenkins.

How to Setup a Staging Wordpress Server

Tue, 18 May 2021 00:00:00 +0000

Having a staging server for Wordpress is quite useful for testing new features without the worry of breaking something in production. It is also helpful when you have a multi user environment and need to test something that could impact others. In this article it is assumed that your Wordpress instance has SELinux enabled as is the case on RHEL/CentOS for example. On other distros like Ubuntu or Debian you can omit the steps specific to SELinux.

Dynamic partitioning in Foreman kickstart templates

Wed, 12 May 2021 00:00:00 +0000

Foreman’s provisioning system allows creating provisioning templates with highly dynamic content. This article describes how to create dynamic content in your Foreman provisioning templates. While the example is related to dynamic partitioning in kickstart, the basic mechanisms for setting this up applies to any kind of template.

Foreman provisioning templates are written in ERB format. This allows having logic in the template itself based on user-defined host parameters. For example the Kickstart default template consumes a bunch of host parameters that allow you to define what kind of kickstart file you want Foreman to spit out for each host or host group. To give a simple example there’s the “lang” parameter which sets the value for “lang” kickstart option:

Fixing foreman-installer "Forward DNS points to x" errors

Wed, 03 Mar 2021 00:00:00 +0000

This posting comes strictly from the land of esoteria. We have a Vagrant + Virtualbox environment that sets up Foreman used for PXE booting baremetal servers. The environment work fine on Linux and MacOS. But we needed to make this environment work also on Windows. Easy, right? Vagrant and Virtualbox are supposed to abstract all those pesky operating system settings away, correct? After resolving a non-Windows related challenge provisioning of Foreman on Windows almost worked. Except that it failed when foreman-installer attempted to start[*]. The error message was this:

Solving an Apache Mellon redirect loop mystery

Sat, 27 Feb 2021 00:00:00 +0000

If you’re at all like me, you every now and then find yourself thrown out of your comfort zone, when you should actually be in it.

The pattern usually goes something like this:

It’s something simple. I’ll fix it in a couple of minutes and document it for others. I know my stuff.
Hmm, this seems to be a bit stubborn. I need to take a closer look.
Still doesn’t work. I need to run this in a debugging set up and break it into pieces, to see where the problem is.

Authenticate and authorize Grafana users from FreeIPA groups via Keycloak

Thu, 28 Jan 2021 00:00:00 +0000

Grafana is a common tool to visualize data from multiple datasources. Perhaps the most common datasource is Prometheus. If an organization has a Single-Sign On solution, it makes sense to authenticate users centrally with that solution That will make authentication easier and friendlier for end users (authenticate once and then access multiple services), and also enable stronger authentication by adding multifactor authentication.

In addition to authenticating user, it is also possible to authorize Grafana users based on their group membership.

Making sense of JBOSS/Wildfly interfaces in domain mode

Thu, 28 Jan 2021 00:00:00 +0000

In this blog we consider JBoss/Wildfly domain mode in the context of the wonderful Keycloak software. It is not necessarily trivial to understand how the interfaces should be configured, especially if you want to do something other than the defaults, for example to secure your Wildfly/JBOSS configuration, or if you are dealing with a more complex nodes with multiple network interfaces.

Assumptions:

We want to use Keycloak in domain mode. In this operating mode Keycloak application is clustered and also the caches, like user sessions are replicated in the cluster.

High-level and low-level tasks in Kanban

Sun, 17 Jan 2021 00:00:00 +0000

We use Kanban to coordinate our projects and tasks. Kanban is focused on visualizing and improving throughput, that is, making projects (high-level) and tasks (low-level) go through the system as fast as by limiting work in progress, which in turn minimizes waste (time and money).

Our field of work is IT operations and we’re generally quite busy with fulfilling requests from our internal and external stakeholders. Most of our tasks are quite small, about 1-4 hours each, and produce value immediately. We can generally get these done (“through the system”) quickly without having to stop in the middle and can then forget about them. However, we also have bigger tasks that tend to take a lot longer, maybe 8-24 hours of focused work and often involve multiple persons.

Keycloak, Wildfly and Infinispan caches

Wed, 16 Dec 2020 00:00:00 +0000

We recently hit an interesting issue with an “almost in production” Keycloak high-availability clustered domain mode setup that was behind an Amazon EC2 load balancer with sticky sessions enabled. The symptom was that changes from the domain controller did not propagate to the slave. This included:

Changes made to realms and clients using the Admin REST API
User sessions

This lead us into a quest to truly understand how Infinispan caches in the Wildfly application server and consequently in Keycloak work. Before you read this article you should probably read all of these:

Managing Keycloak configuration programmatically

Fri, 11 Dec 2020 00:00:00 +0000

Keycloak is an excellent Open Source Identity and Access Management solution that builds on top of the Wildfly application server. We manage several Keycloak installations for providing SSO with FreeIPA credentials for self-hosted and SaaS services via SAML and OIDC.

To keep our Keycloak configurations healthy and in a known-good state we manage their configurations programmatically. As we use Puppet for our configuration management needs we’ve contributed a lot to the treydock/puppet-module-keycloak project. There is also a Terraform Keycloak provider that covers the Keycloak API very well, but it won’t help you one bit with setting up Keycloak initially. This can be problematic if you’re setting up something more complex like clustered domain mode with TLS reverse proxies.

Puppet types and providers development part 3: on resource uniqueness

Sat, 25 Apr 2020 00:00:00 +0000

This blog post is a part of this blog post series:

In the previous blog post we talked about how to create a resource, a LibreNMS service, in the provider implementation. This blog post will talk about identifying a resource that is being managed. This is a requirement for modifying existing resources.

Modifying existing resources can actually be more tricky than it seems, because you first need to figure out if the resource Puppet is managing already exists on the target system. In many cases the resource type you’re managing has an attribute - for example “name”, “hostname”, “fqdn”, “ip” - that is guaranteed to be unique at the target system and works well as the identifier at the Puppet end. This is the case, for example with files (fully-qualified path), packages (name of the package) and in our particular case, LibreNMS devices (hostname).

Puppet types and providers development part 4: caching resource properties to improve performance

Sat, 25 Apr 2020 00:00:00 +0000

This blog post is a part of this blog post series:

In the previous blog post we talked about how to identify the resources that are being managed. This post will talk about the various ways of fetching the current state of the resources you wish to manage.

Puppet manages a resource’s properties by checking their current state and if the current state differs from the state defined in the Puppet manifest (“desired state”) then Puppet will execute the commands necessary to fix the situation. So, when writing a Puppet provider you need to figure out a way to get the current state of all the resources of the type (e.g. file, package, LibreNMS service) you’re managing as efficiently as possible.

Integrating Puppet Bolt with PuppetDB

Fri, 24 Apr 2020 00:00:00 +0000

In the Puppet Bolt with a Puppet control repository blog post we showed how to integrate Puppet Bolt with an existing control repository. We used a static inventory file, which is ok for testing or when you have a few target nodes. However, if you have a Puppetserver running and are not using Bolt to manage the configurations of your nodes, then the chances are that you have a fairly large node population and maintaining a static inventory would be both tedious and error-prone. One option among many is to use the PuppetDB plugin to generate a dynamic inventory and that is what we describe here.

Puppet types and providers development part 2: creating a resource

Sat, 07 Mar 2020 00:00:00 +0000

This blog post is a part of this blog post series:

In the previous blog post we created the Puppet type librenms_service and created a dummy skeleton for the provider implementation. We were able to use the type, but it did not yet do anything. In this blog post we move to implementing the create method, which in this case was the low-hanging fruit. At this point we do not need to worry about any existence checks (the exists? method) or trying to minimizing the number of API calls with self.prefetch or something similar.

Puppet types and providers development part 1: creating the type

Tue, 03 Mar 2020 00:00:00 +0000

This article is a part of this blog post series:

Writing Puppet types and providers looks like black magic even to those who have lots of experience in the Puppet language itself. While the official types development documentation is quite ok, provider documentation is definitely lacking. Not only are there no practical examples from real life, the important features such as prefetching, resource methods and flushing are described only on a very high level. The usual answer to lack of examples is “have a look at the built-in resources”. But when you look at any built-in resource, you realize that the code is very complex because

Pidgin IRC with ZNC and Freenode SASL external auth

Thu, 12 Dec 2019 00:00:00 +0000

I use Pidgin as my instant messaging client because it is open source and because it supports almost all IM protocols directly or through plugins. In my case I need support for IRC, XMPP, Facebook, Slack and SIPE (Office 365). Apparently it would be possible to use Pidgin for Signal as well, but as official Signal client is open source and works quite well I’m fine with running it standalone. As I use Linux, I can rarely hope for a native client for the proprietary IM services. Plus even if there was one, I would not want to run 4-5 messaging applications when I can have one (or two). Or to have the same amount of IM web applications open just to get messages.