502 Bad gateway when communicating between docker containers in the same network

Question

I am trying to setup grafana with prometheus, cadvisor and alarmmanager. I'm trying to do this via docker-compose. I have got everything working except for grafana to commmunicate with prometheus as a datasource. First my setup via docker-compose.yml

version: '3.7'

volumes:
prometheus_data: {}
grafana_data: {}

networks:
monitor-net:

services:
prometheus:
  image: prom/prometheus:v2.13.1
  volumes:
    - ./prometheus/:/etc/prometheus/
    - prometheus_data:/prometheus
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--web.console.libraries=/usr/share/prometheus/console_libraries'
    - '--web.console.templates=/usr/share/prometheus/consoles'
  ports:
    - 9090:9090
  expose:
    - 9090
  networks:
    - monitor-net
  links:
    - cadvisor:cadvisor
    - node-exporter:node-exporter
  restart: always

node-exporter:
  image: prom/node-exporter
  volumes:
    - /proc:/host/proc:ro
    - /sys:/host/sys:ro
    - /:/rootfs:ro
  command:
    - '--path.procfs=/host/proc'
    - '--path.sysfs=/host/sys'
    - --collector.filesystem.ignored-mount-points
    - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/net>    ports:
    - 9100:9100
  expose:
    - 9100
  networks:
    - monitor-net
  restart: always

alertmanager:
  image: prom/alertmanager
  ports:
    - 9093:9093
  expose:
    - 9093
  volumes:
    - ./alertmanager/:/etc/alertmanager/
  networks:
    - monitor-net
  restart: always
  command:
    - '--config.file=/etc/alertmanager/config.yml'
    - '--storage.path=/alertmanager'

cadvisor:
  image: google/cadvisor
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
  ports:
    - 8080:8080
  expose:
    - 8080
  networks:
    - monitor-net
  restart: always

grafana:
  image: grafana/grafana:6.7.2
  user: "472"
  ports:
    - 3000:3000
  expose:
    - 3000
  volumes:
    - grafana_data:/var/lib/grafana
    - ./grafana/provisioning/:/etc/grafana/provisioning/
  env_file:
    - ./grafana/config.monitoring
  networks:
    - monitor-net
  links:
    - prometheus:prometheus
  restart: always

I can access al services individually in the browser via a domain name coupled to my VPS. I can also see in portainer they really are all 6 in the same network. But when it try to access http://prometheus:9090 from my grafana application:

i get the following error:

t=2021-03-21T19:24:09+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/21/api/v1/query remote_addr=172.26.0.1 referer=http://<my-remote-host>:3000/datasources/edit/21/ error="http: proxy error: dial tcp 172.26.0.5:9090: i/o timeout"


t=2021-03-21T19:24:09+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/21/api/v1/query status=502 remote_addr=172.26.0.1 time_ms=30061 size=0 referer=http://<my-remote-host>:3000/datasources/edit/21/

Now i have successfully done similar setups of containers communicating with each other but i can't get this to work.. I can't open up the port in UFW(running ubuntu server) for obvious security reasons, but that shouldn't make a difference. I also got iptables disabled in the docker-daemon.json

I hope you guys might have a suggestion!