0

I have this architecture in GCP.

Network project: 2 VPC, dev and pro, with two subnets each one. Subnet A is in europe-west1 and subnet B is in us-central1. The same in pro VPC. Both VPC are shared to other projects and I have a GKE cluster on each other project; so I have one GKE cluster in subnet A in VPC dev, one GKE cluster in subnet B in VPC dev and the same in pro VPC.

I have other project(project B) with some tools like Grafana, vault etc with other vpc and a subnet in europe-west1 which I have connected to dev VPC by VPC peering. In this project, I have GKE clusters too. My issue is, if I want to connect from project B to any GKE cluster in dev VPC, if I try to connect with subnet A works fine but with subnet B it doesn't and I don't know why. I have tried creating firewall rules but it doesn't work, even with firewall rules that allow all the traffic and all the ports.

Edit:

Now I can ping from GKE cluster in project B to a pod or node in subnet B but I can't access an Internal IP load balancer in subnet B (I have the same service in subnet A and it works fine). I'm trying with firewall rules that allows all the in/out traffic and all the ports but still not working

My architecture are this one:

enter image description here

My issue now is I can curl (for example) from GKE cluster 5 to an internal Load Balancer service in GKE cluster 1 and 3 but I can't to GKE cluster 2 and 4.

This is my service, it works because from another pod inside the cluster, curl works

apiVersion: v1
kind: Service
metadata:
  annotations:
    networking.gke.io/load-balancer-type: Internal
  name: prometheus
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 9090
    protocol: TCP
    targetPort: 9090
  selector:
    app: prometheus
    prometheus: prometheus-kube-prometheus-prometheus
  sessionAffinity: None
  type: LoadBalancer

Edit: I uploaded the image from network tests:

enter image description here

Thanks

EMG
  • 51
  • 6
  • 1
    set up a VPC Network Peering between the two networks and then set up routes that direct traffic to the appropriate subnet. Additionally, you may need to create firewall rules in both networks to allow the traffic to flow through. For example, you could create a firewall rule to allow traffic from Project B's subnet to the dev VPC Subnet B. You could also create a firewall rule to allow traffic from the dev VPC Subnet B to Project B's subnet. Finally, you may need to configure the network settings on your GKE clusters to enable the traffic to flow. – Hemanth Kumar Feb 02 '23 at 11:42
  • I have checked the GKE cluster config and are the same in network and security. I can see in the vpc peering the routes are ok and I already tried to create some firewall rules that allows all the traffic and ports just to test it and the same issue, I can connect to subnet A but not to subnet B – EMG Feb 02 '23 at 12:05
  • Can you have a check as VPC sharing is not possible with private mail id and org mail id is necessary as it involves IAM. Refer to this [VPC peering restrictions](https://cloud.google.com/vpc/docs/using-vpc-peering#restrictions) – Hemanth Kumar Feb 02 '23 at 12:13
  • I create a new firewall rule and now I can ping one GKE node but I can't access by curl to a service on 9090 port – EMG Feb 02 '23 at 12:27
  • 1
    Refer this [doc](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules) Adding firewall rules for specific use cases and also refer to this [SO](https://stackoverflow.com/questions/21065922/) on How to open a specific port such as 9090. – Hemanth Kumar Feb 02 '23 at 12:41
  • I will check the doc. Also, I have a rule that allow all traffic and ports and doesn't connect but to subnet A it does. It's so strange – EMG Feb 02 '23 at 14:12
  • Can you share with me the architecture diagram or network topology diagram so that it could be helpful in addressing the issue and also from where you are trying to access the internal Load Balancer IP. – Hemanth Kumar Feb 03 '23 at 09:50
  • 1
    I uploaded a picture, hope was clear – EMG Feb 03 '23 at 10:23
  • As per official [doc](https://cloud.google.com/vpc/docs/vpc-peering#interaction-dynamic-routing-mode) If dynamic routing mode of the exporting VPC network is regional, then that network exports dynamic routes only in the same region. if your network is global then only GKE5 will interact with GKE2 and GKE4 as both are not in same region. Please check this [doc](https://cloud.google.com/network-intelligence-center/docs/connectivity-tests/concepts/overview) for some connectivity test and may help in resolving your issue. – Hemanth Kumar Feb 03 '23 at 14:09
  • I set the three VPC as global (they were in regional) but doesn't work anyway – EMG Feb 06 '23 at 10:19
  • I think my issue is networks of different regions cannot connect, but I don't know why because the vpc are global now – EMG Feb 06 '23 at 10:44

2 Answers2

0

set up a VPC Network Peering between the two networks and then set up routes that direct traffic to the appropriate subnet. Additionally, you may need to create firewall rules in both networks to allow the traffic to flow through. For example, you could create a firewall rule to allow traffic from Project B's subnet to the dev VPC Subnet B. You could also create a firewall rule to allow traffic from the dev VPC Subnet B to Project B's subnet. Finally, you may need to configure the network settings on your GKE clusters to enable the traffic to flow.

Can you have a check as VPC sharing is not possible with private mail id and org mail id is necessary as it involves IAM. Refer to this VPC peering restrictions

Refer this doc Adding firewall rules for specific use cases and also refer to this SO on How to open a specific port such as 9090.

As per official doc If dynamic routing mode of the exporting VPC network is regional, then that network exports dynamic routes only in the same region. if your network is global then only GKE5 will interact with GKE2 and GKE4 as both are not in same region.

Please check this doc for some connectivity test and try to test the connectivity this will arrest the possibility that the firewall/ routes issue

Hemanth Kumar
  • 2,728
  • 1
  • 4
  • 19
  • I'm checking the routes but, I can see routes are automatically created when I create the peering, I hadn't create anyone – EMG Feb 08 '23 at 12:57
  • we got your point but may i know whether u have run connectivity tests etc.. also having VM inplace of GKE(in all the involved projects) to check the connection also helps to isolate GKE config related issue – Hemanth Kumar Feb 09 '23 at 10:20
  • I did a network test between the ips and it crashes in forwarding rule. I have updated the post with a picture. – EMG Feb 09 '23 at 14:29
0

Finally I found the solution. The Load Balancer needs this annotation:

networking.gke.io/internal-load-balancer-allow-global-access: "true"

I tried it and it works

Thanks to all

EMG
  • 51
  • 6