Access Control for the Prometheus Pushgateway

Question

We have a Prometheus Pushgateway running and listening to metrics push from our AWS Lambda function. However, the URL to the Pushgateway is accessible by the public, which might impose some security issues. We were wondering if there is any way we could add a layer of protection to the Pushgateway so that it is not publicly accessible?

I found this Github thread that may answered this question: https://github.com/prometheus/pushgateway/issues/281

It proposed to set up a reverse proxy in front of the pushgateway. However, I am still confused on how that may actually work? We are currently using Kubernetes to deploy the Prometheus.

score 2 · Accepted Answer · answered Jan 30 '20 at 20:24

2

You can include authentication in your ingress controller by using a TLS secret as an ingress rule. Here's an example that shows how to generate basic auth for your ingress:

https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

Also, don't forget to include the Python handler function in your client to set the auth header as pointed out here:

https://github.com/prometheus/client_python#handlers-for-authentication

answered Jan 30 '20 at 20:24

Sunny

46
3

1

Thanks for the solution, I followed the document and it worked perfectly! – RonZhang724 Jan 30 '20 at 21:20

score 1 · Answer 2 · answered Jan 29 '20 at 13:44

1

A suggestion here will be to make the URL of the Pushgateway Internal by using an AWS Internal Load Balancer, create an AWS Private Hosted Zone attach your VPC to this zone after this the next step will be to deploy the lambda in the same VPC.

This should solve the security issue.

answered Jan 29 '20 at 13:44

gsood

86
2

Thanks for the answer, this is a good solution but I am personally more familiar with the ingress controller configuration. But still, I appreciate your effort. Upvoted! – RonZhang724 Jan 30 '20 at 21:21

score 1 · Answer 3 · answered Feb 17 '22 at 22:39

You are right, you need reverse proxy here. I also faced with the same issue, so you need nginx in front of your prometheus/pushgateway.

First, install nginx using this article (you can start from Step 8 — Securing Prometheus if you already configured prometheus):

My nginx config :

events { }
http {
upstream prometheus {
      server 127.0.0.1:9090;
      keepalive 64;
}

upstream pushgateway {
      server 127.0.0.1:9091;
      keepalive 64;
}

server {
      root /var/www/example;
      listen 0.0.0.0:80;
      server_name __;      
      location / {
            auth_basic "Prometheus server authentication2";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_pass http://prometheus;
      }  
}


server {
      root /var/www/example;
      listen 0.0.0.0:3001;
      server_name __;      
      location / {
            auth_basic "Pushgateway server authentication";
            auth_basic_user_file /etc/nginx/.htpasswd;
            proxy_pass http://pushgateway;
      } 
}
}

my pushgateway.service file :

[Unit]
Description=Pushgateway
Wants=network-online.target
After=network-online.target

[Service]
User=pushgateway
Group=pushgateway
Type=simple
ExecStart=/usr/local/bin/pushgateway --web.listen-address="127.0.0.1:9091" --web.telemetry-path="/metrics"  --persistence.file="/tmp/metric.store"  --persistence.interval=5m --log.level="info" --log.format="logger:stdout?json=true"

[Install]
WantedBy=multi-user.target

It is important to set : --web.listen-address="127.0.0.1:9091", not ":9091" - so it will be exposed only to localhost.

Through the nginx pushgateway will be accessible on port 3001, port 9091 will be not public. Base authentication will be required to have access or push metrics.

About how to test it using Postman you can find here

Access Control for the Prometheus Pushgateway

3 Answers3