Making Prometheus and Alertmanager work behind a reverse proxy
We maintain a Prometheus and Alertmanager setup where both services are running on the same host and an Apache reverse proxy handles TLS and authentication, authorization and single sign-on with Apache mod_auth_mellon configured as a Keycloak SAML client. This way we can limit access to Prometheus and Alertmanager to people who belong to a certain FreeIPA group.
From end-user perspective Prometheus and Alertmanager are available at these locations:
In this setup both Prometheus and Alertmanager require additional startup options to work correctly. First Prometheus:
-
--web.listen-address=“127.0.0.1:9090”
-
--web.external-url=“https://prometheus.example.org/prometheus/"
-
--web.route-prefix=”/"
Then Alertmanager:
-
--web.listen-address=“127.0.0.1:9093”
-
--web.external-url=“https://prometheus.example.org/alertmanager/"
-
--web.route-prefix=”/"
This configuration was inspired by this comment in a GitHub issue. The fact that you need both --web.external-url and --web.route-prefix is actually mentioned in some official examples (e.g. here). Nevertheless there are several GitHub issues (e.g. #1583, #2193, #4295) discussing this topic, which means people find this topic confusing.
With the above Prometheus and Alertmanager settings you can get away with a very basic reverse proxy setup without any rewriting rules. The mod_auth_mellon setup is a different topic and won’t be covered in this blog post.