We recently hit an interesting issue with an “almost in production” Keycloak high-availability clustered domain mode setup that was behind an Amazon EC2 load balancer with sticky sessions enabled. The symptom was that changes from the domain controller did not propagate to the slave. This included:

• Changes made to realms and clients using the Admin REST API

• User sessions

This lead us into a quest to truly understand how Infinispan caches in the Wildfly application server and consequently in Keycloak work. Before you read this article you should probably read all of these:

Our company is about infrastructure automation and products built on top of it. That means we really don’t want to do manually what a computer can do. Therefore it was not an option to publish our blog posts first in Wordpress, then copy-and-paste them to LinkedIn. Rather, let a Wordpress plugin configured as a LinkedIn app do the hard word. So, from now on all of our blog posts should get published automatically on our “Puppeteers Oy” LinkedIn page. Let’s hope that the automation does not break too often :).

We - like many others - have the problem that our work gets easily interrupted. Some interruptions come from the inside. For example, you remember “a thing you need to do” and without thinking start doing it immediately. Other interruptions are external, for example a colleague ask you for advise or a customer calls you asking for help. Almost always these things could wait a bit, but the person simply acted on an impulse.

Keycloak is an excellent Open Source Identity and Access Management solution that builds on top of the Wildfly application server. We manage several Keycloak installations for providing SSO with FreeIPA credentials for self-hosted and SaaS services via SAML and OIDC.

To keep our Keycloak configurations healthy and in a known-good state we manage their configurations programmatically. As we use Puppet for our configuration management needs we’ve contributed a lot to the treydock/puppet-module-keycloak project. There is also a Terraform Keycloak provider that covers the Keycloak API very well, but it won’t help you one bit with setting up Keycloak initially. This can be problematic if you’re setting up something more complex like clustered domain mode with TLS reverse proxies.

We maintain a Prometheus and Alertmanager setup where both services are running on the same host and an Apache reverse proxy handles TLS and authentication, authorization and single sign-on with Apache mod_auth_mellon configured as a Keycloak SAML client. This way we can limit access to Prometheus and Alertmanager to people who belong to a certain FreeIPA group.

From end-user perspective Prometheus and Alertmanager are available at these locations:

• https://prometheus.example.org/prometheus

• https://prometheus.example.org/alertmanager

In this setup both Prometheus and Alertmanager require additional startup options to work correctly. First Prometheus:

Separating data from code in Puppet modules is advisable as it improves reusability of code. The separation can be accomplished with Hiera by having separate levels based on facts, organizational units, locations, etc.

Hiera can also be used for storing private data that needs to be protected and must not be readable by outsiders. Typically the data, like all other code, is stored in a version control system that may be located outside the organisation. In such a case Hiera encryption is a must.

If the workflow that includes the control repository, r10k and GitLab still feels too light, there are yet more ways to make the workflow heavier by adding more abstraction to the Puppet modules themselves. This is done by adopting the “Roles and Profiles pattern” that was developed by Craig Dunn and popularized in the blog posts by Gary Larizza.

Roles and profiles are just Puppet classes. Profiles are usually used for creating technology stacks. A good example is a web application whose installation requires a web server (nginx, apache), a database server (postgresql, mysql) and several other components. The configurations for such an application could of course be done with a “regular” Puppet module but the reusability of the module would be rather limited for two reasons:

In the previous post I discussed the control repository structure. In this post I’ll talk about r10k. That tool is used for deploying control repository branches into matching Puppet environments on the Puppet server. Installing r10k is very straightforward with the bundled gem:

$ /opt/puppetlabs/puppet/bin/gem install r10k

After this you’ll need a settings file for r10k, /etc/puppetlabs/r10k/r10k.yaml:

--- 
 :cachedir: /opt/puppetlabs/puppet/cache/r10k 
 :sources: 
 control: 
 basedir: /etc/puppetlabs/code/environments 
 prefix: false 
 remote: git@gitlab.com:myorg/control-repo.git

Several requirements must be met before the configuration above will work:

In the previous posts of this series I discussed the control repository and r10k. The last component in fattening the workflow is Git provider such as GitLab, GitHub or Bitbucket. Nowadays all of them are reasonable choices for storing private Git modules, which Puppet control repositories and site-specific modules tend to be. One of the biggest advantages of GitLab over its competition is that it is open source and can be installed locally.

If editing Puppet code and Hiera directly with puppetmaster feels too easy, you can complicate the Puppet workflow as much you like by adding more components to the palette. Adding the so called control repository and r10k to Puppet environment maintenance instantly adds several phases to your initially simple workflow. There are also benefits that I’ll discuss later on.

The control repository (control repo for short) is a Git repository that is used to create dynamic Puppet environments on the Puppet master. Each Git branch can be turned into a matching Puppet environment. The name of the default branch, production, is the same as the environment the Puppet agents use by default. The content of the control repo branches look like this: