Making Prometheus and Alertmanager work behind a reverse proxy

December 9, 2020 – Samuli Seppänen

We maintain a Prometheus and Alertmanager setup where both services are running on the same host and an Apache reverse proxy handles TLS and authentication, authorization and single sign-on with Apache mod_auth_mellon configured as a Keycloak SAML client. This way we can limit access to Prometheus and Alertmanager to people who belong to a certain FreeIPA group.

From end-user perspective Prometheus and Alertmanager are available at these locations:

  • https://prometheus.example.org/prometheus
  • https://prometheus.example.org/alertmanager

In this setup both Prometheus and Alertmanager require additional startup options to work correctly. First Prometheus:

  • --web.listen-address="127.0.0.1:9090"
  • --web.external-url="https://prometheus.example.org/prometheus/"
  • --web.route-prefix="/"

Then Alertmanager:

  • --web.listen-address="127.0.0.1:9093"
  • --web.external-url="https://prometheus.example.org/alertmanager/"
  • --web.route-prefix="/"

This configuration was inspired by this comment in a GitHub issue. The fact that you need both --web.external-url and --web.route-prefix is actually mentioned in some official examples (e.g. here). Nevertheless there are several GitHub issues (e.g. #1583, #2193, #4295) discussing this topic, which means people find this topic confusing.

With the above Prometheus and Alertmanager settings you can get away with a very basic reverse proxy setup without any rewriting rules. The mod_auth_mellon setup is a different topic and won't be covered in this blog post.

Want to talk to an expert?

If you want to reach us, just send us a message or book a free call!
menucross-circle