4

I have hard time getting this working with NLB using ingress controller : https://kubernetes.github.io/ingress-nginx/deploy/#network-load-balancer-nlb

Even subnets are not taking effect here , its not passing my configurations in the API that creates the NLB:

================================
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-eip-allocations: "eipalloc- 
    07e3afcd4b7b5d644,eipalloc-0d9cb0154be5ab55d,eipalloc-0e4e5ec3df81aa3ea"
    service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet- 
    061f4a497621a7179,subnet-001c2e5df9cc93960"
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
suren
  • 7,817
  • 1
  • 30
  • 51
amit
  • 88
  • 1
  • 5
  • 1
    Can you please add more details? What steps have you followed? Can you add the output of this command and possible logs if you see pending state or any other problem? `$ kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch` – Mark Watney Dec 11 '19 at 12:50
  • 1
    Are you talking about public IP address for ingress controller's service? How your K8S cluster was installed(Managed service EKS, kubeadm, kubespray, kops...)? Could you show the output of ```kubectl -n get svc ingress-nginx ingress-nginx?``` – Oles Rid Dec 11 '19 at 15:05
  • yes i am talking about public ip – amit Dec 12 '19 at 11:11
  • How did you get Allocation IDs? – Melissa Jenner Oct 09 '20 at 19:58

2 Answers2

3

The number of eip allocations must match the number of subnets in the subnet annotation.

service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-xyz, eipalloc-zzz

service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-xxxx, mySubnet

You have 3 allocations but only 2 subnets.

In addition, the annotation

service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing" is missing.

By default this will use scheme "internal".

I assume since you are allocating elastic IP addresses that you might want "internet-facing".

Also, you are using annotations that are meant for "AWS Load Balancer Controller" but you are using an "AWS cloud provider load balancer controller"

The external value for aws-load-balancer-type is what causes the AWS Load Balancer Controller, rather than the AWS cloud provider load balancer controller, to create the Network Load Balancer. docs

You are using service.beta.kubernetes.io/aws-load-balancer-type: nlb which means that none of the links provided earlier in this answer pertain to your Load Balancer. nlb type is an "AWS cloud provider load balancer controller" not an "AWS Load Balancer Controller"

For "AWS cloud provider load balancer controller" all the docs reference is this.

ComradeJoecool
  • 734
  • 6
  • 18
1

So, as it turned out - these annotations will be supported only since Kubernetes 1.16, which is "coming soon" on AWS. Currently supported version is 1.15, which just ignores those annotations...

Considering that you are using AWS-specific annotations here (service.beta.kubernetes.io/aws-load-balancer-eip-allocations) - I assume that this is exactly the reason why it does not work on your case.

As a workaround, I would advice:

  1. Create custom post-deployment script that re-configures newly-created LoadBalancer, after each Kubernetes Service Update.
  2. Switch to use something more conventional, like ELB with your Container, and AutoScaling groups (that's what we did.)
  3. Setup your own Kubernetes Controller (super-hard thingie, which will become completely obsolete and will just be basically a lost of time, as soon as 1.16 is officially out). See this how-to
  4. Wait...

Official statement: https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html#1-16-prequisites

Full list of annotations (when they will be "supported" ofc): https://github.com/kubernetes/kubernetes/blob/v1.16.0/staging/src/k8s.io/legacy-cloud-providers/aws/aws.go#L208-L211

Stay tuned! :(

Der Zinger
  • 506
  • 7
  • 13
  • Just for historical reasons, short update: We've managed to get it working on `1.17`, but be wary that provisioning eIPs will take some noticeable time (up to 10 minutes in our case). Also, in case you really need similar functionality - you could consider using `Terraform`, which doesn't really complicates the setup much, while providing you options for post-deployment actions, such as assigning those IPs, while keeping your setup more version-agnostic. – Der Zinger Dec 13 '20 at 04:55