Introduction
This documentation is related to 'Prometheus Image', created by Puppeteers Oy. Prometheus Image is an open source, pre-built Cloud virtual machine image that runs Prometheus, Alertmanager, Node Exporter and Blackbox Exporter. Additional exporters, namely Cloudwatch Exporter and Azure Metrics Exporter have been preinstalled but are not active by default. All these services run as systemd units for your convenience.
Currently Prometheus Image is available for Azure Marketplace and AWS Marketplace. You can choose between Ubuntu 24.04 and AlmaLinux 9 base operating systems.
Prometheus is a cloud-native metrics-gathering and monitoring system. It is most commonly paired with Alertmanager for alerting. It has a smaller footprint than most SNMP-based solutions and is more flexible and easier to extend. All configurations are stored in configuration files and there is no database to manage, migrate or back up. For data visualization Prometheus is often connected with Grafana.
Security group setup
Prometheus Image has services running on the following ports:
- Prometheus (tcp/9090)
- Alertmanager (tcp/9093)
- Node Exporter (tcp/9100)
- Blackbox exporter (tcp/9115)
The following services are not enabled by default as they require extra configuration:
- Azure Metrics Exporter (tcp/8080)
- Cloudwatch Exporter (tcp/9106)
None of the ports need nor should be exposed to the Internet except to validate that they work. We recommend opening these ports only to applicable RFC1918 private networks. Even a better option is to limit access to Prometheus and Alertmanager to your VPN and only expose the exporter endpoints to the Prometheus server, is by default on the same host.
Security recommendations
In production setups we strongly recommend allowing access to all Prometheus services only from RFC1918 private networks. If you don't have a VPN (e.g. OpenVPN, OpenVPN Access Server or Wireguard) configured and must access a production Prometheus services directly from the Internet, it is recommended for security reasons to limit access using IP whitelisting in Security Groups.
Note that Prometheus, Alertmanager and all the various exporters don't support HTTPS or authenticating users. While data in Prometheus does not very security sensitive, exposing Prometheus to the Internet opens up lots of potential vectors of exploit. So, please disable direct access from the Internet immediately after you have verified that the product works.
Logging in to the instance using SSH
Our images are built on top of official Cloud images of various operating system vendors. In AWS login as the "ubuntu" (Ubuntu) or "ec2-user" (AlmaLinux). In Azure you'd typically log in as "azureuser". You can login using the instance's public IP address.
Note that automated system package upgrade may kick in immediately after you launch the instance. This may manifest itself as slowness in the web interface which should not last very long.
Accessing Prometheus services
Prometheus Image exposes the following services by default:
- Prometheus: http://<ip>:9090
- Alertmanager: http://<ip>:9093
- Node Exporter: http://<ip>:9100
- Blackbox exporter:http://<ip>:9115
If you configure authentication and other details for the optional exporters they will be available as well:
- Azure Metrics Exporter: http://<ip>:8080
- Cloudwatch Exporter: http://<ip>:9106
No login or authentication is required to access these services. Enabling HTTPS and authentication will require further configuration, which may include things like setting up reverse proxies, basic authentication, SAML authentication or similar. If you need help with post-install setup please consider subscribing to a Premium plan which includes support services.
List of systemd services
The Prometheus Image has the following systemd services:
- prometheus
- alertmanager
- node_exporter
- blackbox_exporter
- cloudwatch-exporter
- azure_metrics_exporter
Setting the hostname
On production installations it is recommended to change the hostname of the instance to something reasonable. On Ubuntu and AlmaLinux you can do that like this:
$ sudo -i
$ hostnamectl set-hostname prometheus.example.orgUsing the Prometheus Image
Once the product is running, you can verify that it is working by navigating to the following URLs:
- Prometheus at http://<ip>:9090
- Should show one active alert called Watchdog on the Alerts tab
- All other alerts should show green
- Alertmanager at http://<ip>:9093
- Should show one active alert
- Node Exporter at http://<ip>:9100/metrics
- Should return a long list of metrics
- Blackbox Exporter at http://<ip>:9115/metrics
- Should return a long list of metrics
In AWS you can login with SSH as "ubuntu" (Ubuntu image) or "ec2-user" (AlmaLinux image). In Azure you'd typically login as "azureuser".
Once logged in you can check the bundled but inactive-by-default services:
$ sudo -s
$ systemctl status cloudwatch-exporter.service
$ systemctl status azure_metrics_exporter.serviceBoth services should be "disabled" and "stopped". You must configure them before enabling them.
You can modify the configuration of the bundled components by editing their respective configuration files under /etc as root. After each configuration change restart the applicable systemd service.
Getting started with Prometheus
Prometheus has a fairly steep learning curve, but once you get past that it is like wax in your hands. SNMP-based systems often focus on the current metrics and alert if some thresholds are reached. Prometheus, on the other hand, is all about gathering time-based metrics from various sources. What you do with those metrics is your business.
In Prometheus you craft queries in PromQL language and get results back. It is up to you to determine if the results necessitate sending an alert with Alertmanager. One of the strengths of Prometheus is that you can alert on trends instead of reacting to the metrics as they stand right now.. So, for example you could alert one month in advance before a disk is going to be full, given historic data on disk consumption. Or you can ignore CPU usage spikes by looking at the average over a longer period of time.
With Alertmanager you configure very complex alert routing based on severity of the problem, how critical the monitored service and who is responsible for it. Such flexibility is not common in SNMP-based monitoring solutions.
Most of the time Prometheus gets its metrics from so-called exporters. Some of the more common exporters are bundled with Prometheus Image:
- Node exporter: collects metrics about the machine - virtual or baremetal - it runs on. Used to monitor things such as disk space, CPU usage, memory usage and systemd services. It supports gathering metrics from arbitrary scripts using the Textfile collector.
- Blackbox exporter: probe network endpoints over HTTP, HTTPS, DNS, TCP, ICMP and gRPC. Used to detect crashed services, certificate validity monitoring and such.
- Cloudwatch exporter: collects metrics from AWS Cloudwatch.
- Azure metrics exporter: collects metrics from Azure Monitor
The Prometheus project contains a comprehensive list of available exporters.
The best way to get started with Prometheus is to read the official documentation. When using Prometheus Image you can safely skip the installation and configuration instruction and start from the Expression Browser documentation:
- https://prometheus.io/docs/prometheus/latest/getting_started/#using-the-expression-browser
For alerting check the Alertmanager documentation. When you use Prometheus Image you can skip the Alertmanager install and configuration steps.
Contact and support
Please contact our support at [email protected] if have any problems deploying Prometheus Image or with the default configuration of the Prometheus Image on a freshly created cloud instance. We will respond on a best effort basis.
We do not provide support for running or configuring Prometheus for your particular use-case, unless you have a Premium plan (only available in Azure). If you need without a Premium plan we recommend reaching out to the official community support channels.
