We recently wrote a Nagios plugin for getting metrics from AWS Cloudwatch to LibreNMS. As the customer was ok with releasing it as open source here it is. It is written in Ruby - we like the language and we’re also Puppet guys, so it felt like a natural choice for us.

The plugin allows getting as many metrics you want from Cloudwatch and setting alert thresholds (warning, critical) for each metric statistic (e.g. “average CPUUtilization percent”) separately. In many cases it is not useful to configure alerts for a metric (e.g. “minimum CPUUtilization percent”), but you do want to graph it regardless - this plugins allows that.

As mentioned in our earlier blog post we created an “email to XMPP bridge” called milter-xmpp for forwarding alert emails from monitoring systems to an XMPP chatroom. It was known that if the connection to the XMPP server ever died the bridge would stop working, but we wanted to know how fast that would happen. It took some days, but eventually the connection died.

The fix ended up being quite trivial, but we needed a way to break the connection. Fortunately the good folks at StackExchange already had a good solution: use the “ss” tool (“another utility to investigate sockets”) which provides output similar to netstat:

We have used XMPP (Jabber) for our internal discussions for years. In the recent years it seems that XMPP has lost some of its traction. For example, Google and Facebook used to use XMPP for instant message and Slack used to have an XMPP bridge in their service. All of those are now gone and have been replaced with proprietary messaging protocols. In the context of a big company having all the control over the protocol probably makes sense, but it of course makes life difficult for users who have to have a separate client for each or have to use a multi-protocol client such Pidgin just to keep their sanity. And none of the big players seems to “winning the war” for users.

Over the years I’ve had to automate installation of Ubuntu and Debian installers a number of times. The primary motivation has always been to avoid all the questions about timezone, locale, partitioning and which packages to install and to not have to babysit the installation. In theory automating the installation is as simple as writing a preseed file. In practice it is much harder for a number of reasons:

• Documentation on the Internet around custom installation media and preseeding setup is anecdotal at best - it gets you going and then lets you fall on your face. The official documentation is pretty good, though.
• The exact format of the preseed questions varies by operating system version, not just by operating system (Debian, Ubuntu).
• In most cases questions about locale and keyboard layout must have answers before the installer preseed is even loaded. This is misleading, because the preseed file does have those questions, even though the answers have zero practical effect for the installer.
• There are many ways to accomplish preseeding, most notably by creating a custom installation medium with the preseed file, or by getting the preseed from an URL (“auto mode”).
• The default keyboard layout in most Ubuntu/Debian installer is en-US. This is fine, except when you don’t have an American keyboard: then you will be struggling to find characters such as “:” or “/”, which are relatively essential when loading the preseed file from an URL.
• It is impossible to remember how to start “automated install” from a vanilla installer menu, let alone figure out what the keyboard and locale settings should look like on the kernel command-line.
• The installer is happy to produce unworking configurations (e.g. no bootloader or no kernel) without giving any hints why things are not working.
• Each mistake is costly in time, especially if it affects something like Grub configuration which happens at the end of the install.
• Because of above, you tend to try to make as many changes to the kernel command line and/or preseeding as possible in one go. This means that when you get things working you don’t know what the proper fix was, and don’t feel like repeating the process to find out.

In order not to be completely negative here is the general recipe for success:

Our puppet-snmpd module support SNMPv3 very well across a wide range of operating systems. Because o that SNMPv3 is what we use across the board in multiple environments. However, in the rare cases when I have to debug SNMP problems I can never remember the exact snmpwalk command-lines to use. So, here is the usual way to run snmpwalk to figure out if SNMP is providing data:

$ snmpwalk -Os -v 3 -u monitor -l authPriv -a sha -A <auth-pass> -x AES -X <encryption-pass> hostname

Run from the monitoring server (e.g. LibreNMS configured with puppet-librenms) to make sure basic connectivity and access controls are correct as well. Or against localhost if you’re interested in the data and not connectivity.

Järjestimme Turku Business Regionin kanssa yhteistyössä webinaarin 5.5.2020 aiheesta “Infrastruktuurin rakehtaminen koodilla”. Webinaaritallenne on saatavilla täällä:

• Infrastruktuurin rakentaminen koodilla

Webinaarissa käsiteltiin perusasioita seuraavista aiheista: infrastruktuurin rakentaminen koodilla eli IaC, versionhallint, laadunhallintamenetelmät sekä työkalut kuten Puppet, Terraform, Ansible sekä Puppet Bolt.

Tässä webinaaritallenteessa esittelemme perusteet infrastruktuurin rakentamisesta koodilla, mukaan luettuna versionhallinnan, laadunvarmistuksen ja erilaiset työkalut kuten Puppetin, Terraformin, Ansiblen ja Puppet Boltin:

• Infrastruktuurin rakentaminen koodilla

Toteutettu yhteistyössä Turku Business Regionin kanssa 5.5.2020.

This blog post is a part of this blog post series:

In the previous blog post we talked about how to create a resource, a LibreNMS service, in the provider implementation. This blog post will talk about identifying a resource that is being managed. This is a requirement for modifying existing resources.

Modifying existing resources can actually be more tricky than it seems, because you first need to figure out if the resource Puppet is managing already exists on the target system. In many cases the resource type you’re managing has an attribute - for example “name”, “hostname”, “fqdn”, “ip” - that is guaranteed to be unique at the target system and works well as the identifier at the Puppet end. This is the case, for example with files (fully-qualified path), packages (name of the package) and in our particular case, LibreNMS devices (hostname).

This blog post is a part of this blog post series:

In the previous blog post we talked about how to identify the resources that are being managed. This post will talk about the various ways of fetching the current state of the resources you wish to manage.

Puppet manages a resource’s properties by checking their current state and if the current state differs from the state defined in the Puppet manifest (“desired state”) then Puppet will execute the commands necessary to fix the situation. So, when writing a Puppet provider you need to figure out a way to get the current state of all the resources of the type (e.g. file, package, LibreNMS service) you’re managing as efficiently as possible.

In the Puppet Bolt with a Puppet control repository blog post we showed how to integrate Puppet Bolt with an existing control repository. We used a static inventory file, which is ok for testing or when you have a few target nodes. However, if you have a Puppetserver running and are not using Bolt to manage the configurations of your nodes, then the chances are that you have a fairly large node population and maintaining a static inventory would be both tedious and error-prone. One option among many is to use the PuppetDB plugin to generate a dynamic inventory and that is what we describe here.