Monitoring Azure with Prometheus

October 31, 2024

Introduction

Prometheus is a great tool for “cloud-native” metrics gathering and monitoring. It is particularly easy to manage with infrastructure as code methods. Prometheus can collect metrics from various sources, including Azure Monitor. You can use Azure Metrics Exporter to convert Azure Monitor metrics into Prometheus metrics, but doing so is definitely not trivial unless you’re pretty familiar with Prometheus, Azure Monitor and Microsoft Entra ID apps. This article tries to help you make monitoring Azure with Prometheus a breeze instead of a struggle. We won’t go into a lot of detail about Prometheus or Azure Monitor as there’s plenty of information available elsewhere about both of them. Also note that this article is about vanilla Prometheus, not Microsoft’s “Azure Monitor managed service for Prometheus”.

Proof of concept with Azure CLI

First verify that you can get metrics from Azure Monitor with Azure CLI. First login to Azure from a shell:

az login

If you have access to multiple Microsoft Entra ID directories use the --tenant parameter to select the one Prometheus should monitor. We assume here that your used has the necessary permissions to read data from Azure Monitor.

Now let’s query some data from Azure. You can query metrics for any Azure resource, but here we use a virtual machine as an example:

az monitor metrics list --resource test-vm --resource-group test-rg --resource-type Microsoft.Compute/virtualMachines --metrics "Percentage CPU"
{
  "cost": 59,
  "interval": "PT1M",
  "namespace": "Microsoft.Compute/virtualMachines",
  "resourceregion": "northeurope",
  "timespan": "2024-10-30T16:52:58Z/2024-10-30T17:52:58Z",
  "value": [
    {
      "displayDescription": "The percentage of allocated compute units that are currently in use by the Virtual Machine(s)",
      "errorCode": "Success",
      "id": "/subscriptions/bcdef012-3456-789a-bcde-f0123456789a/resourceGroups/test-rg/providers/Microsoft.Compute/virtualMachines/test-vm/providers/Microsoft.Insights/metrics/Percentage CPU",
      "name": {
        "localizedValue": "Percentage CPU",
        "value": "Percentage CPU"
      },
      "resourceGroup": "test-rg",
      "timeseries": [
        {
          "data": [
            {
              "average": 0.43,
              "timeStamp": "2024-10-30T16:52:00Z"
            },
            {
              "average": 0.435,
              "timeStamp": "2024-10-30T16:53:00Z"
            },
            {
              "average": 0.435,
              "timeStamp": "2024-10-30T16:54:00Z"
            },
            --- snip ---
          ],
          "metadatavalues": []
        }
      ],
      "type": "Microsoft.Insights/metrics",
      "unit": "Percent"
    }
  ]
}

This test proves that your Microsoft Entra ID user can download metrics from Azure Monitor. The next step is to create an “app”, in practice an oauth2 client, in Microsoft Entra ID.

Create a Microsoft Entra ID app for authentication

It is generally not a good idea to use a real person’s Microsoft Entra ID credentials for applications such as Azure Metrics Exporter. Moreover, if MFA is enabled in Microsoft Entra ID, then this sort of credentials misuse becomes almost impossible. So, we create an Entra ID “app” that we can use as a service principal to autenticate with Azure:

Go to Microsoft Entra ID and select App registration
- The name can be azure-metrics-exporter or similar
- Create a new secret for the app and set the expiration to (e.g. 2 years)
  - Get the Value of the secret immediately; it is only available when you create it
- Get the Application (client) ID for the app
Go to the Subscription that has the resource you wish to monitor and select Access Control (IAM)
- Add a new Role assignment
- Select Monitoring Reader as the role
- Select azure-metrics-exporter as a member

This Application (client) ID and Value (of the secret) will later be used to authenticate Azure Metrics Exporter with Azure.

Building Azure Metrics Exporter

The Azure Metrics Exporter project does not provide any executables. So you need to build the azure-metrics-exporter binary yourself with Go. You have two options there which we describe below.

Option 1: using podman-builder to build an RPM

We used our own podman-builder framework to build Azure Metrics Exporter and package it as rpm. We use the same framework to build and package Prometheus, Alertmanager and various other exporters. All of those packages include systemd unit files which make deployment to baremetal and virtual machines very easy.

The podman-builder framework is mainly tested on Fedora Linux, but it is known to work on Ubuntu 24.04. It depends on Pod m an, which you think of as “Docker from Red Hat that sucks a lot less”. Almost all the skills youve acquired with Docker can be mapped 1:1 to Podman, as the latter aims to be as compatible with the former as possible.

The build process for Azure Metrics Exporter with podman-builder is quite trivial. You should run the build as a normal user (not as root):

podman volume create podman-builds
git clone git@github.com:Puppet-Finland/podman-builder.git
cd podman-builder
./build.sh -p azure-metrics-exporter -o rocky-9

If all goes well, you will find an RPM in ~/.local/share/containers/storage/volumes/podman-builds/_data/rocky-9/azure-metrics-exporter. Also, if you are so inclined, you can get just the azure-metrics-exporter from under the target directory in there.

Option 2: building manually

Installing latest Go binary release

First make sure you have the latest Go binary release installed. If you’re running Linux the installation process might look like this:

wget --quiet https://go.dev/dl/go1.23.2.linux-amd64.tar.gz
tar -zxf go1.23.2.linux-amd64.tar.gz
mv go /usr/local

You may then need to set GO environment variables. For example:

export GOOS=linux
export GOARCH=amd64
export GOROOT=/usr/local/go

Building Azure Metrics Exporter

Finally you can build Azure Metrics Exporter, in this case version 24.9.1:

git clone https://github.com/webdevops/azure-metrics-exporter.git
cd azure-metrics-exporter
git checkout 24.9.1
make build

If all goes well the build will produce an azure-metrics-exporter executable which you can the deploy.

Creating a systemd unit file

If you are going to run azure-metrics-exporter on a baremetal or virtual machine you want to have a systemd unit file for it. Here’s the one that podman-builder generates and which seems to work nicely:

[Unit]
Description=Azure Metrics Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=azure-metrics-exporter
Group=azure-metrics-exporter
ExecStart=/usr/local/bin/azure-metrics-exporter \
--server.bind=:8080 \
--enable-caching
EnvironmentFile=/etc/azure-metrics-exporter/azure-metrics-exporter.env
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

The EnvironmentFile will contain the Azure credentials (client id and secret). The parameters passed to the binary are described in Azure Metrics Exporter documentation.

Configuring Azure Metrics Exporter authentication

Azure Metrics Exporter keeps most of its configuration in the main Prometheus configuration file. However, Azure credentials are fetched from environment variables, probably to better support container deployments and for generic “it makes sense from security perspective” reasons. The systemd unit file is pointed to an environment file, /etc/azure-metrics-exporter/azure-metrics-exporter.env, which looks like this:

# Environment file for azure-metrics-exporter
#
# This file is used by azure-metrics-exporter systemd service to authenticate
# with Azure. Typically you would create an Azure "app" (i.e. Oauth2 client)
# and credentials for it and use those to authenticate. The easiest way is to
# use a client secret, although certificate authentication is also a
# possibility.
#
# For details take a look here:
#
# https://github.com/webdevops/go-common/blob/main/azuresdk/README.md

# App (oauth2 client) ID
AZURE_CLIENT_ID=01234567-89ab-cdef-0123456789abcdef0

# App (oauth2 client) secret
AZURE_CLIENT_SECRET=long-random-string

# Azure tenant (domain)
AZURE_TENANT_ID=acme.inc

# Azure subscriptions to target (comma- or space-separated)
AZURE_SERVICEDISCOVERY_SUBSCRIPTION_ID=bcdef012-3456-789a-bcde-f0123456789a

The AZURE_CLIENT_ID must match the Application (client) ID from the Microsoft Entra ID app you created earlier. Similarly, AZURE_CLIENT_SECRET must match the Value of the secret you created for the app. The AZURE_TENANT_ID must match the name of the Microsoft Entra tenant that has the Azure resources you wish to monitor. The AZURE_SERVICEDISCOVERY_SUBSCRIPTION_ID should match Subscription you chose when granting the app Monitoring Read permissions.

In a containerized deployment you would pass these environment variables directly to the runtime environment.

Testing Azure Metrics Exporter with curl

Once Azure Metrics Exporter runs you might be tempted to just start monitoring Azure with Prometheus and be done with it. I advise doing so, because you first should understand how to scrape metrics from Azure Metrics Exporter and how it generates metric names.

What we do below is query the /probe/metrics/list HTTP endpoint and passing it some parameters. The parameters are documented on the Azure Metrics Exporter home page. The curl incantation is long, but not particularly complex. Unlike earlier in this blog post we get the “VMAvailabilityMetric” which basically outputs 1 if the VM is up:

curl -G --data-urlencode "subscription=bcdef012-3456-789a-bcde-f0123456789a" --data-urlencode "name=azure_metric_vmavailability" --data-urlencode "resourceType=Microsoft.Compute/virtualMachines" --data-urlencode "metric=VMAvailabilityMetric" --data-urlencode "interval=PT5M" --data-urlencode "timespan=PT5M" --data-urlencode "aggregation=minimum" http://localhost:8080/probe/metrics/list

# HELP azure_metric_vmavailability Azure monitor insight metric
# TYPE azure_metric_vmavailability gauge
azure_metric_vmavailability{aggregation="minimum",interval="PT5M",metric="VmAvailabilityMetric",resourceGroup="test-rg",resourceID="/subscriptions/bcdef012-3456-789a-bcde-f0123456789a/resourcegroups/test-rg/providers/microsoft.compute/virtualmachines/test-vm",resourceName="test-vm",subscriptionID="bcdef012-3456-789a-bcde-f0123456789a",subscriptionName="Microsoft Azure Sponsorship",tag_owner="",timespan="PT5M",unit="Count"} 1
--- snip ---

Now that we know that Azure Metrics Exporter actually works, we can finally create a Prometheus job for it.

Monitoring Azure with Prometheus

With all the proof of concepts behind us we can finally move to monitoring Azure with Prometheus. We basically just convert the curl incantation into a Prometheus job. Some things are worth noting though:

All the entries under metric need to be valid Azure Monitor metric names.
The name parameter becomes the basename of the metric - or rather a group of metrics - in Prometheus. The metric label tells which Azure Monitor metric is being tracked.

So, you could for example query all the VM metrics you want in one job and create alert rules for each Azure Monitor metric separately by using the metric label.

Here’s a sample Prometheus job that replicates what we did with curl above:

- job_name: azure_metrics_vm
  scrape_interval: 60s
  metrics_path: "/probe/metrics/list"
  params:
    name:
    - azure_metric_vmavailability
    subscription:
    - bcdef012-3456-789a-bcde-f0123456789a
    resourceType:
    - Microsoft.Compute/virtualMachines
    metric:
    - VMAvailabilityMetric
    interval:
    - PT5M
    timespan:
    - PT5M
    aggregation:
    - minimum
  static_configs:
  - targets:
    - localhost:8080

That’s it. After reloading/restarting Prometheus you should start receiving metrics from Azure Monitor in your Prometheus instance.

NOTE: if you have any issues following this article please don’t hesitate to reach out to us.

Deep-Dive