3

I have implemented a multi master HA kubernetes cluster and wanted to implement the Calico the hardway as described in here. I was able complete all the steps and my connectivity is not there between the pods and services and pods and other pods in different nodes.

only, difference is I use two different AZs in AWS and I suppose it should not be an issue. I can see pods are getting the IPs and calico network interfaces are creating but still see the connectivity as I explained. Node even doesn't have the public internet access. I did the BGP configuration exactly same in the guide but no luck and I'm not quite sure something to be changed in the BGP configuration when it comes to multi-AZ deployment. I'm not much aware of the Calico BGP configuration.

Unfortunately, calicoctl node diags does not properly run and not providing much more information to move forward.

I'd love here your valuable thoughts and constructive criticism to fix this.

Aruna Fernando
  • 101
  • 7
  • Can you try to enable `“CrossSubnet” IPIP` mode as described [here](https://docs.projectcalico.org/reference/public-cloud/aws) ? – acid_fuji Nov 09 '20 at 08:20
  • Actually, I found this after posting this and I followed the with IPIP "CrossSubnet" and "natOutgoing" true now ended up with something different. Which you can find it [here](https://github.com/projectcalico/calico/issues/4150): I was clue less and want to fix this in order to verify the functionality of this. – Aruna Fernando Nov 10 '20 at 09:05
  • I`ll have a look. Just to be sure, have you also disabled [src/dest checks](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck) for ec2 instances? – acid_fuji Nov 10 '20 at 09:23
  • Yes, I did that too :( – Aruna Fernando Nov 10 '20 at 10:51
  • After careful looking at the github issue I think it would best to have separate question asked about your next problem. This is how SO recommend to deal with the question. Another problem means another question. (check [How to ask a good question](https://stackoverflow.com/help/how-to-ask)). Since my suggestion helped you moving forward I`ll prepare an answer that would be helpful to anyone else face similar issues. – acid_fuji Nov 10 '20 at 11:31
  • 1
    Appreciate your valuable thoughts, you mean ask in here Stackoverflow? I was thinking giving it a try again while chasing my github issue. If it is better. I can raise another one. Thanks for the help! – Aruna Fernando Nov 10 '20 at 15:42
  • BTW, Thomas - I was able to run Calico installation without an issue and easily following [this](https://docs.projectcalico.org/getting-started/kubernetes/self-managed-onprem/onpremises#install-calico-with-kubernetes-api-datastore-more-than-50-nodes) with the same subnet, But it was not my intention. It seems something need to be changed or adjust according to my environment. The question is how, I'm clueless. But planning to give another go and I will update the status here. – Aruna Fernando Nov 10 '20 at 15:46
  • Sure thing. Git it ago here or Severfault if you don`t have any replies on github. Once you will decide to do that plesase do describe well your environment. That would speed the help you need from others. – acid_fuji Nov 10 '20 at 15:53
  • @aruna-lakmal Any solutuion for the above issue? Currently i'm facing same issues, pod to pod comminication fails and pods to internet also. – pbms May 14 '21 at 18:33

1 Answers1

2

Calico configured in BGP mode requires all of the instances to be located in the same subnet to work out of the box.

To use calico with deployments that are split across multiple availability zones you must:

Disable AWS source / destination check (see here):

You can do that using AWS CLI:

    aws ec2 modify-instance-attribute --no-source-dest-check --instance-id          $EC2_INSTANCE_ID --region <REGION-WHERE-EC2-INSTANCE-IS-LAUNCHED>

Or using the AWS console:

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
  2. In the navigation pane, choose Instances.
  3. Select the NAT instance, choose Actions, Networking, Change Source/Dest. Check.
  4. For the NAT instance, verify that this attribute is disabled. Otherwise, choose Yes, Disable.
  5. If the NAT instance has a secondary network interface, choose it from Network interfaces on the Description tab and choose the interface ID to go to the network interfaces page. Choose Actions, Change Source/Dest. Check, disable the setting, and choose Save.*

Enable IPIP encapsulation and outgoing NAT on your Calico IP pools

(IPPool) represents a collection of IP addresses from which Calico expects endpoint IPs to be assigned. (see here how to set it up)

, then all of the Kubernetes instances must be located in the same subnet for Calico to work out of the box.

To enable the “CrossSubnet” IPIP feature, configure your Calico IP pool resources to enable IPIP mode to “CrossSubnet” like in the example below:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: ippool-multi-az
spec:
  cidr: 192.168.0.0/16
  ipipMode: CrossSubnet
EOF

Example above refers to AWS cloud configuration taken from the Calico documentation. Please note that Calico docs has also information about GCP, Azure and IBM.

Remark: If you face another problems going "the hard way" you may want to use as a reference another cluster created by following calico guides below:

Lastly, it is worth to check is also this very good document about calico routing modes (it shows also cross subnets ipip mode).

acid_fuji
  • 6,287
  • 7
  • 22
  • 1
    Thanks thomas, I hope this will help others to get the things kick-off. I will share the feedback after testing again or if I don't hear anything back from the Github issue. +1 from me. cheers!!! – Aruna Fernando Nov 11 '20 at 04:20