Installing Prometheus with SELinux

Introduction

In the world of serverless, there are still system admins who host applications on ec2 instances, while most of them are Amazon Linux they use the same package manager as RHEL, CentOS that is yum or DNF

There are many tutorials on how to set these up, but setting these up with SELinux disabled is not recommended. There are alternatives to SELinux such as Systemd Sandboxing. Most of the tutorials tell you to straight-up disable SELinux, so in this one, I'm asking you to be a bit patient and troubleshoot.

One of the best SELinux tutorial/Debug guide is here and here

Getting your hands dirty!

Firstly, let's create a Prometheus user.

# useradd -s /sbin/false prometheus
or
$ sudo useradd -s /sbin/false prometheus

Install the SELinux Tools listed here

To start, install and extract the Prometheus zip file from

sudo -u prometheus  wget https://github.com/prometheus/prometheus/releases/download/v2.35.0/prometheus-2.35.0.linux-amd64.tar.gz

Then unzip it.

sudo -u prometheus tar -xvf prometheus-2.35.0.linux-amd64.tar.gz

Once you create the prometheus service, vim /etc/systemd/system/prometheus.service

# /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/home/prometheus/prometheus \
--config.file=/home/prometheus/prometheus.yml \
--storage.tsdb.path=/home/prometheus/ \
--web.console.templates=/home/prometheus/consoles \
--web.console.libraries=/home/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Then do:

# systemctl daemon-reload
# systemctl start prometheus

You will get hit with an error if you check journalctl -t setroubleshoot

SELinux is preventing /usr/lib/systemd/systemd from execute access on the file prometheus.

*****  Plugin catchall (100. confidence) suggests   **************************

If you believe that systemd should be allowed execute access on the prometheus file by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c '(ometheus)' --raw | audit2allow -M my-ometheus
# semodule -X 300 -i my-ometheus.pp


Additional Information:
Source Context                system_u:system_r:init_t:s0
Target Context                unconfined_u:object_r:admin_home_t:s0
Target Objects                prometheus [ file ]
Source                        (ometheus)
Source Path                   /usr/lib/systemd/systemd
Port                          <Unknown>
Host                          rocky8.5
Source RPM Packages           systemd-239-51.el8_5.5.x86_64
Target RPM Packages
SELinux Policy RPM            selinux-policy-targeted-3.14.3-80.el8_5.2.noarch
Local Policy RPM              selinux-policy-targeted-3.14.3-80.el8_5.2.noarch
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Host Name                     rocky8.5
Platform                      Linux rocky8.5 4.18.0-348.23.1.el8_5.x86_64 #1 SMP
                              Wed Apr 27 15:32:52 UTC 2022 x86_64 x86_64
Alert Count                   1
First Seen                    2022-05-06 21:43:51 IST
Last Seen                     2022-05-06 21:43:51 IST
Local ID                      562818dd-247b-40b2-9b24-f63020bb46b7

At this point, you're probably going, "Oh shit it's SELinux again, I should disable it", nope.

The error has too much info, so I just focus on Source Context and Target Context as you can see, prometheus is a binary and systemd is trying to run it, but it fails.

SELinux uses domains, so when you try to execute a binary systemd has to transition to that domain, for further reading I would suggest Gentoo's wiki

This can be checked by searching the domains using:

sesearch --allow --source init_t --target user_home_t  --class file --perm execute

This will yield no results. However, this will:

sesearch --allow --source init_t --target bin_t --class file --perm execute
allow init_t base_ro_file_type:file { execute execute_no_trans getattr ioctl lock map open read };
allow init_t bin_t:file { execute execute_no_trans getattr ioctl lock map open read }

As you can see here, the source (domain) init_t can perform { execute execute_no_trans getattr ioctl lock map open read }; on the target domain bin_t:file of the type file. More detailed information can be found here

A simple fix is to relabel the library using a file that you know works

chcon --reference=/bin/less /home/prometheus/prometheus

This will fix the issue with the systemd service, however it is prone to a relabel on system reboot. To make it permanent

semanage fcontext -a -t bin_t "/home/prometheus/prometheus"

That's it, yes SELinux is complicated, and hard, but tutorials such as the ones on Gentoo make it easier, even I'm quite new to it and learning it, so if I've made mistake please correct me!

I can be found @mediocredevops on Twitter, or LinkedIn.

If you've enjoyed it or have any queries, please feel free to reach out.

Resources

Gentoo Wiki RedHat SELinux