Prometheus on Puppeteers Oy

Monitoring Azure with Prometheus

Thu, 31 Oct 2024 00:00:00 +0000

Introduction

Prometheus is a great tool for “cloud-native” metrics gathering and monitoring. It is particularly easy to manage with infrastructure as code methods. Prometheus can collect metrics from various sources, including Azure Monitor. You can use Azure Metrics Exporter to convert Azure Monitor metrics into Prometheus metrics, but doing so is definitely not trivial unless you’re pretty familiar with Prometheus, Azure Monitor and Microsoft Entra ID apps. This article tries to help you make monitoring Azure with Prometheus a breeze instead of a struggle. We won’t go into a lot of detail about Prometheus or Azure Monitor as there’s plenty of information available elsewhere about both of them. Also note that this article is about vanilla Prometheus, not Microsoft’s “Azure Monitor managed service for Prometheus”.

Blackbox Exporter ICMP module and selinux

Thu, 21 Sep 2023 00:00:00 +0000

Blackbox Exporter ICMP module and selinux do not like each other. Same goes for systemd units and ICMP Echo sockets. However, with persistence, skill and a bit of luck you can make it all work together. Photo credit.

Introduction

Blackbox Exporter is an exporter for the Prometheus monitoring system. It is able to probe a target, such as a host or an application from the outside. Blackbox Exporter does probing with modules such as http, tcp, dns, icmp and grpc (details here). The icmp module is special because it does privileged operations that require the CAP_NET_RAW capability and as well as custom selinux policies. This article aims to explain how you can make Blackbox Exporter ICMP module and selinux work together. NOTE: you do not need any of these tricks if you use non-icmp Blackbox exporter modules.

Excluding down targets in Prometheus

Fri, 11 Aug 2023 00:00:00 +0000

Excluding down targets in Prometheus is easy. It allow you to monitor volatile systems that start and shut down on demand.

Introduction

Prometheus is an open source metrics and alerting solution used to monitor a wide range of things. Unlike many classic network monitoring systems Prometheus is focused on colleting metrics. This makes it quite easy to do things that would be difficult in others systems. For example excluding down targets in Prometheus is quite trivial. This can be highly useful when you have volatile, static targets that come and go - for example Jenkins slaves (or build agents). Even on such targets you’d want to monitor things like systemd service status, memory usage and disk usage.

Integrating Twilio SMS and Prometheus Alertmanager

Thu, 16 Feb 2023 00:00:00 +0000

Introduction

Prometheus is a widely used, cloud-native open source monitoring solution. Its alerting component is called Alertmanager, which can send alerts to email, Slack and elsewhere. Both Prometheus and Alertmanager fit very well into the infrastructure as code model as well. Twilio SMS is a SMS sending service with consumption-based pricing model with an extensive API. With some effort you can integrate Twilio SMS and Prometheus together as described in this article.

Testing AlertManager webhooks with curl

Mon, 06 Feb 2023 00:00:00 +0000

Introduction

Prometheus is a Cloud-native metrics platforms that is very easy to manage with infrastructure as code tools. Prometheus is often couple with Alertmanager which handles alerting and alert routing. AlertManager has good support for various alert transport (e.g. email or slack) but its alerting capabilities can be extended with custom webhooks. When AlertManager is configured to use a webhook in an alert route, it forwards alerts to an HTTPS/HTTP endpoint instead of handling transporting of the alert itself. AlertManager’s official documentation gives a rough idea of what kind of payload to use and how to configure the webhook in alertmanager.yaml. What is not documented is how you can send test alerts to the webhook without having to trigger real (or fake) alerts in Prometheus. While using fake alerts does work, it is a slow, tedious and invasive process. This article shows one way for testing AlertManager webhooks with curl. In our case the target was to speed up testing of changes to promtotwilio.

Making Prometheus and Alertmanager work behind a reverse proxy

Wed, 09 Dec 2020 00:00:00 +0000

We maintain a Prometheus and Alertmanager setup where both services are running on the same host and an Apache reverse proxy handles TLS and authentication, authorization and single sign-on with Apache mod_auth_mellon configured as a Keycloak SAML client. This way we can limit access to Prometheus and Alertmanager to people who belong to a certain FreeIPA group.

From end-user perspective Prometheus and Alertmanager are available at these locations:

In this setup both Prometheus and Alertmanager require additional startup options to work correctly. First Prometheus: