Web Socket wss:// in AWS EKS with nginx Ingress. returns 400. ws://, http:// and https:// works perfectly

Question

I've been trying to run few services in AWS EKS Cluster. I followed the ingress-nginx guide to get https with AWS ACM certificate

https://kubernetes.github.io/ingress-nginx/deploy/#aws

Used tls termination at ingress controller

I used 3 routes for each services as

adminer.xxxx.com - points to an adminer service

socket.xxxx.com - points to the wss service written in nodejs

service.xxxx.com - points to a program that returns a page which connects to socket url

Without TLS Termination, in http:// everything works fine, ws://socket.xxxx.com/socket.io gets connected and responds well.

When I add TLS, the request goes to wss://socket.xxxx.com/socket.io and the nginx returns 400. I Can't figure out why it happens.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations: 
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 100m
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_set_header Host $http_host;
    # nginx.ingress.kuberenetes.io/use-regex: "true"
spec:
  rules:
    - host: adminer.xxxx.com
      http:
        paths:
          - path: /
            backend:
              serviceName: adminer-svc
              servicePort: 8080
    - host: socket.xxxx.com
      http:
        paths:
          - path: /
            backend:
              serviceName: nodejs-svc
              servicePort: 2020
    - host: service.xxxx.com
      http:
        paths:
          - path: /
            backend:
              serviceName: django-svc
              servicePort: 8000

I Tried with and without these configurations

nginx.ingress.kubernetes.io/configuration-snippet: |
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection $connection_upgrade;
          proxy_set_header Host $http_host;

Also I've tried changing the socket.xxxx.com into service.xxxx.com and assigned to be forwarded for /socket.io path

I've also put a url in nodejs with express to test if its working at all, and it responds properly in https://

Only the wss:// has the issue.

PS : This entire Service works when nginx is setup in a normal system with nginx configuration

location / {
      proxy_pass http://localhost:2020/;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;
}

I tried request like this as well

https://node-socket.xxxx.com/socket.io/?EIO=3&transport=polling this works

https://node-socket.xxxx.comsocket.io/?EIO=3&transport=websocket this doesnt.

Combinations I tried

protocol, balancer, backendproto, transport => result
wss://, ELB, TCP, websocket  =>  400
wss://, NLB, TCP, websocket  =>  400
wss://, ELB, HTTP, websocket  =>  400
wss://, NLB, HTTP, websocket  =>  400
ws://, ELB, TCP, websocket => 400
ws://, ELB, HTTP, websocket => 400
ws://, NLB, TCP, websocket => 400
ws://, NLB, HTTP, websocket => 400

polling worked in every cases

check the logs of nginx ingress controller to check possible causes of 400 response code. — Tarun Khosla, Jul 28 '20 at 11:11
Check if this helps https://stackoverflow.com/questions/58534215/kubernetes-ingress-websockets-connection-issue — Tarun Khosla, Jul 28 '20 at 11:12
I found out the 400 is from nginx controller while checking the logs. There is no further explainations. Just bad request. checked via kubectl logs, should I check anywhere else ? — Adharsh M, Jul 28 '20 at 11:20
I don't have the certificate files to add like that, I'm using aws ACM i just have an ARN to add. — Adharsh M, Jul 28 '20 at 11:21
I think there is related[ github issue](https://github.com/kubernetes/ingress-nginx/issues/3746) about that, take a look. To be more precisely check this [comment](https://github.com/kubernetes/ingress-nginx/issues/3746#issuecomment-474507823). So it should work if you terminate ssl on nginx side. — Jakub, Jul 29 '20 at 09:04
I've already tried that @jt97 but my other services running in http not working when i changed that -backend-protocol: "tcp" — Adharsh M, Jul 29 '20 at 15:31
Did you try making two different Ingress resource , one with `service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"` for your current issue and one without this for other routes ? — Tarun Khosla, Jul 31 '20 at 14:18
@TarunKhosla absolutely , I tried 1) wss:// tcp elb websocket 2) wss:// tcp nlb websocket 3) wss:// http elb websocket 4) wss:// http nlb websocket nothing worked, but polling works — Adharsh M, Aug 02 '20 at 00:58

score 1 · Answer 1 · answered Apr 07 '22 at 23:57

1

You seems to be missing

nginx.org/websocket-services

annotation

It's value should be a value of kubernetes service name. See https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

answered Apr 07 '22 at 23:57

Piotr

317
1
13

score 0 · Answer 2 · answered Oct 27 '22 at 18:29

0

@pitor, both ingresses are different, so annotations will differ.

answered Oct 27 '22 at 18:29

Rajiv Reddy

1
1

This can be just a comment below the question – Borislav Stoilov Nov 02 '22 at 11:32

Web Socket wss:// in AWS EKS with nginx Ingress. returns 400. ws://, http:// and https:// works perfectly

2 Answers2