how to get outside cluster Grafana authentication to get metrics from Prometheus/Thanos

Question

I have a few openshift (4.8) clusters monitored centrally by Thanos.
I'm trying to configure the Grafana (version install outside the clusters 8.3.1) Datasource with no success.

When accessing the URL from my desktop works well. However, when accessing from the Datasource configuration panel I am getting a 403 error

My hypothesis is that I have misconfigured something related to the TLS keys.
Any ideas?

This is the content of the logs error:

t=2021-12-31T16:08:05+0200 lvl=eror msg="Instant query failed" logger=tsdb.prometheus query=1+1 err="Post \"https://thanos-querier-openshift-monitoring.mycluster/api/v1/query\": x509: certificate signed by unknown authority"
t=2021-12-31T16:08:05+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[::1] time_ms=33 size=166 referer=http://localhost:3000/datasources/edit/wTJvnKAnk
t=2021-12-31T16:08:13+0200 lvl=eror msg="Instant query failed" logger=tsdb.prometheus query=1+1 err="client_error: client error: 403"
t=2021-12-31T16:08:13+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[::1] time_ms=53 size=65 referer=http://localhost:3000/datasources/edit/wTJvnKAnk
t=2021-12-31T16:08:14+0200 lvl=eror msg="Instant query failed" logger=tsdb.prometheus query=1+1 err="client_error: client error: 403"
t=2021-12-31T16:08:14+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[::1] time_ms=50 size=65 referer=http://localhost:3000/datasources/edit/wTJvnKAnk
t=2021-12-31T16:39:17+0200 lvl=eror msg="Instant query failed" logger=tsdb.prometheus query=1+1 err="client_error: client error: 403"
t=2021-12-31T16:39:17+0200 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/ds/query status=400 remote_addr=[::1] time_ms=81 size=65 referer=http://localhost:3000/datasources/edit/wTJvnKAnk

Answer 1

You have a problem with TLS verification:

x509: certificate signed by unknown authority

The easiest option is to check Skip TLS Verify in the Prometheus datasource, so Grafana will not verify used TLS certificate used by Prometheus.

More secure (and more complicated) option will be to add used CA cert also to the Grafana container, so Grafana will be able to verify Prometheus cert.