Testing AlertManager webhooks with curl

Introduction

Prometheus is a Cloud-native metrics platforms that is very easy to manage with infrastructure as code tools. Prometheus is often couple with Alertmanager which handles alerting and alert routing. AlertManager has good support for various alert transport (e.g. email or slack) but its alerting capabilities can be extended with custom webhooks. When AlertManager is configured to use a webhook in an alert route, it forwards alerts to an HTTPS/HTTP endpoint instead of handling transporting of the alert itself. AlertManager’s official documentation gives a rough idea of what kind of payload to use and how to configure the webhook in alertmanager.yaml. What is not documented is how you can send test alerts to the webhook without having to trigger real (or fake) alerts in Prometheus. While using fake alerts does work, it is a slow, tedious and invasive process. This article shows one way for testing AlertManager webhooks with curl. In our case the target was to speed up testing of changes to promtotwilio.

Alert payload format

The JSON-formatted alert payload sent by AlertMangaer is documented in the official webhook documentation. However, the sample payload is not usable as-is, so here’s a JSON payload with real data:

{
  "receiver": "sms",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "ProbeFailure",
        "instance": "https://server.example.org",
        "job": "http_checks",
        "monitor": "master",
        "severity": "critical"
      },
      "annotations": {
        "description": "Instance https://server.example.org has been down for over 5m. Job: http_checks",
        "summary": "BlackBox Probe Failure: https://server.example.org"
      },
      "startsAt": "2023-02-06T13:08:45.828Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus.example.org:9090/graph?g0.expr=probe_success+%3D%3D+0\\u0026g0.tab=1",
      "fingerprint": "1a30ba71cca2921f"
    }
  ],
  "groupLabels": {
    "alertname": "ProbeFailure"
  },
  "commonLabels": {
    "alertname": "ProbeFailure",
    "instance": "https://server.example.org",
    "job": "http_checks",
    "monitor": "master",
    "severity": "critical"
  },
  "commonAnnotations": {
    "description": "Instance https://server.example.org has been down for over 5m. Job: http_checks",
    "summary": "BlackBox Probe Failure: https://server.example.org"
  },
  "externalURL": "http://prometheus.example.org:9093",
  "version": "4",
  "groupKey": "{}/{severity=\"critical\"}:{alertname=\"ProbeFailure\"}",
  "truncatedAlerts": 0
}

Save this payload to a file or create your own (see below).

Getting a real alert JSON payload

If your webhook wants to do some fancy stuff the sample data from above may not be enough. Should that be the case, you can trigger a real alert payload from your own environment using tcpdump. Here the webhook service is running on loopback interface on port 9191 on the same host as AlertManager:

tcpdump -A -i lo port 9191

Once tcpdump is running, just trigger a (fake) alert from AlertManager and wait for the JSON payload to appear in ASCII format. You also get the HTTP headers as a bonus, though they should be very similar to the ones shown below. For clarity you may want to pass the JSON payload through a JSON pretty-printing function to make it more readable - it is all in one line by default. Save your payload into JSON file.

This strategy won’t work if traffic between AlertManager and the webhook service is encrypted. If that is the case, you can always make the webhook print out the real AlertManager payloads it receives and get the payload that way.

HTTP header format

Alertmanager sets multiple HTTP headers when connecting to the webhook:

  • Host: 127.0.0.1:9191

  • User-Agent: Alertmanager/0.23.0

  • Content-Type: application/json

Here “Host” refers to the IP and port of the webhook service.

Testing AlertManager webhooks with curl

Once you have the payload and headers you can send a message to the webhook service with curl:

curl \
-i \
-H "Host: 127.0.0.1:9191" \
-H "User-Agent: Alertmanager/0.23.0" \
-H "Content-Type: application/json" \
--request POST \
--data @sample.json \
http://127.0.0.1:9191/send

The path in the webhook service URL is arbitrary. In case of promtotwilio the POST requests must go to /send. Your webhook may behave differently.

The –data option reads a file when the option’s value is prefixed with a “@”. In this case curl reads a file called sample.json and passes it as data.