Making Prometheus and Alertmanager work behind a reverse proxy

December 9, 2020 – Samuli Seppänen

We maintain a Prometheus and Alertmanager setup where both services are running on the same host and an Apache reverse proxy handles TLS and authentication, authorization and single sign-on with Apache mod_auth_mellon configured as a Keycloak SAML client. This way we can limit access to Prometheus and Alertmanager to people who belong to a certain FreeIPA group.

From end-user perspective Prometheus and Alertmanager are available at these locations:

  • https://prometheus.example.org/prometheus
  • https://prometheus.example.org/alertmanager

In this setup both Prometheus and Alertmanager require additional startup options to work correctly. First Prometheus:

  • --web.listen-address="127.0.0.1:9090"
  • --web.external-url="https://prometheus.example.org/prometheus/"
  • --web.route-prefix="/"

Then Alertmanager:

  • --web.listen-address="127.0.0.1:9093"
  • --web.external-url="https://prometheus.example.org/alertmanager/"
  • --web.route-prefix="/"

This configuration was inspired by this comment in a GitHub issue. The fact that you need both --web.external-url and --web.route-prefix is actually mentioned in some official examples (e.g. here). Nevertheless there are several GitHub issues (e.g. #1583, #2193, #4295) discussing this topic, which means people find this topic confusing.

With the above Prometheus and Alertmanager settings you can get away with a very basic reverse proxy setup without any rewriting rules. The mod_auth_mellon setup is a different topic and won't be covered in this blog post.

Did you like the article? Share it with others!
Categories
#aad #Access #acl #alertmanager #ansible #ansible module development #Apache #API #augeas #authentication #authorization #automation #automatization #aws #azure #backup #bash #bitbucket #buildbot #cache #centos #cloud #cloud-init #cloudflare #cloudfront #cluster #connectionsJpa #control repo #custom fact #database #debian #devops #digital sovereignty #DNS #docker #domain mode #duplo #edenred #ejabberd #email #encryption #erb #europe #eyaml #fabric #facter #facts #fargate #fedora #file #finnish #foreman #freeipa #git #github #gitlab #gnome #google #grafana #hammer #hiera #IAM #import #infinispan #Infrastructure as Code #ipmi #irc #jboss #jdk #jenkins #JMESPath #json #kanban #keycloak #letsencrypt #librarian-puppet #librenms #linkedin #Linux #Location #loop #marketing #mautic #Mellon #mfa #microsoft #monitoring #mysql #nagios #network-manager #oauth #oauth2 #office365 #oidc #openshift #open source #openvpn #oxygen #packer #paranormal #pdk #people #php #pkcs7 #pomodoro #Powershell #preseed #presentation #profiles #prometheus #provisioning #puppet #puppet-bolt #puppet-litmus #puppetboard #puppetdb #Puppetfile #puppetserver #puppet types and providers #pxeboot #qemu #quality #r10k #rds #recruitment #redirect #Restrict #Reverse Proxy #robotframework #roles #rspec #ruby #SAML #selinux #sem #shell #showsql #snmp #snmpd #software developement #spam #ssh #sso #standardization #systemd #systemd-resolved #teams #terraform #twilio #ubuntu #user-data #vagrant #vanity awards #variable #vim #virtualbox #visualstudio #webdevelopment #wildfly #Windows #wireguard #wordpress #workflow #x11 #xmpp #zimbra
We are
 Puppeteers
menucross-circle