Blackbox Exporter ICMP module and selinux

Blackbox Exporter ICMP module and selinux do not like each other. Same goes for systemd units and ICMP Echo sockets. However, with persistence, skill and a bit of luck you can make it all work together. Photo credit.

Introduction

Blackbox Exporter is an exporter for the Prometheus monitoring system. It is able to probe a target, such as a host or an application from the outside. Blackbox Exporter does probing with modules such as http, tcp, dns, icmp and grpc (details here). The icmp module is special because it does privileged operations that require the CAP_NET_RAW capability and as well as custom selinux policies. This article aims to explain how you can make Blackbox Exporter ICMP module and selinux work together. NOTE: you do not need any of these tricks if you use non-icmp Blackbox exporter modules.

Our setup was the following

• Blackbox exporter runs on Red Hat Enterprise Linux 8

• Blackbox exporter runs as an unprivileged user

• We launch Blackbox Exporter as a systemd service

• Selinux is in enforcing mode

• Selinux uses the targeted policy

The setup of Prometheus and Blackbox Exporter is outside of the scope of this article. We assume that Blackbox Exporter is working and that you’re able to use, say, the http module to test the availability of a website.

The initial Blackbox Exporter systemd unit file

We assume that you run Blackbox Exporter as a systemd unit file. This will not be the case if you run it as a container with selinux enabled, but the same principles will apply. The initial systemd unit file we used looked like this:

$ systemctl cat blackbox_exporter.service
[Unit]
Description=Prometheus blackbox_exporter
Wants=network-online.target
After=network-online.target

[Service]
User=blackbox-exporter
Group=blackbox-exporter

ExecStart=/usr/local/bin/blackbox_exporter \
--config.file=/etc/blackbox-exporter.yaml

ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

As you can see, this service runs as an unprivileged user and has not specific capabilities.

How to add the CAP_NET_RAW capability

The first step you need to do is grant Blackbox Exporter the CAP_NET_RAW capability. There are several ways to accomplish this.

Option 1: add capabilities to the systemd unit file

This is probably the cleanest and safest way to grant Blackbox Exporter the CAP_NET_RAW capability. You can modify the unit file directly and add the following there:

[Service]
AmbientCapabilities=CAP_NET_RAW

A better option may be to add a systemd unit override file, /etc/systemd/system/blackbox_exporter.service.d/override.conf, with the same content.

In both cases you need to reload the units and restart blackbox exporter.

$ systemctl daemon-reload
$ systemctl restart blackbox_exporter

Option 2: set capabilities directly in the blackbox_exporter executable

Another, somewhat less secure option is to set the capability on the blackbox_exporter executable directly. The problem with this approach is that any user who can start an instance of Blackbox Exporter can use it with CAP_NET_RAW capabilities.

To add the capability to the Blackbox Exporter executable:

$ setcap cap_net_raw+ep /opt/blackbox_exporter-version.linux-amd64/blackbox_exporter

If you want to verify the changes do:

$ getcap /opt/blackbox_exporter-version.linux-amd64/blackbox_exporter 
/opt/blackbox_exporter-version.linux-amd64/blackbox_exporter = cap_net_raw+ep

To remove the capability do:

$ setcap -r /opt/blackbox_exporter-version.linux-amd64/blackbox_exporter

Option 3: set sysctl ping_group_range

The third option is to use sysctl to set the group (GID) range that is allowed to pings, or more correctly, to create ICMP echo sockets. This is a relatively crude method as you can define only one GID range, so adding capabilities is preferable. Moreover if you tend to automate your work with infrastructure as code this kind of system-specific GID ranges are troublesome.

By default a RHEL 8 system only allows root group ping:

$ sysctl net.ipv4.ping_group_range
net.ipv4.ping_group_range = 1   0

To allow Blackbox Exporter to ping you can change this sysctl variable. First check the GID of blackbox-exporter:

$ grep blackbox /etc/group
blackbox-exporter:x:985:

Then adjust the ping_group_range:

sysctl net.ipv4.ping_group_range="0 985"

Now Blackbox exporter should be able to ping without having to set any capabilities.

Making Blackbox Exporter ICMP module and selinux behave

Once you have sorted out capabilities you must tackle selinux. As usual, the problem is that the source context of blackbox-exporter process does not match the target context:

type=AVC msg=audit(1695299538.492:16846): avc:  denied  { node_bind } for  pid=26958 comm="blackbox_export" saddr=127.0.0.1 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:object_r:node_t:s0 tclass=icmp_socket permissive=0

The source type is fairly generic, unconfined_service_t. What this means is that there is no Blackbox Exporter-specific source context. Instead, we create a custom selinux policy that will, effectively, allow other systemd services to create ICMP sockets. Because ICMP socket access is also limited by capabilities that is probably not a big deal in practice.

If you had the exact same selinux issue we had you can use audit2allow to generate a custom policy module:

ausearch -c 'blackbox_export' --raw | audit2allow -M blackbox_exporter

This will produce a human-readable policy file, blackbox_exporter.te:

$ cat blackbox_exporter.te 

module blackbox_exporter 1.0;

require {
        type unconfined_service_t;
        type node_t;
        class icmp_socket node_bind;
}

#============= unconfined_service_t ==============
allow unconfined_service_t node_t:icmp_socket node_bind;

The audit2allow also created a compiled policy module (.pp) which you must now install and activate:

$ semodule -i blackbox_exporter.pp

To verify that the module is installed and active run this:

$ semanage module -l|grep blackbox
blackbox_exporter         400       pp

If you need to remove the module use this:

$ semodule -X 400 -r blackbox_exporter

With this custom policy module Blackbox Exporter ICMP module and selinux should co-exist peacefully.

Blackbox Exporter and Prometheus configuration

There are plenty of other articles that describe how you can configure Blackbox Exporter and ICMP probes. Therefore this section is here mostly for your convenience and for the sake of completeness. First the Blackbox Exporter configuration:

---
modules:
  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: ip4

Then the matching section in Prometheus configuration:

- job_name: icmp_check
  scrape_timeout: 15s
  scrape_interval: 60s
  metrics_path: "/probe"
  params:
    module:
    - icmp
  static_configs:
  - targets:
    - web.example.org
  relabel_configs:
  - source_labels:
    - __address__
    target_label: __param_target
  - source_labels:
    - __param_target
    target_label: instance
  - target_label: __address__
    replacement: localhost:9115

• Deep-Dive

• Prometheus
• Selinux