Kubernetes Websockets using Socket.io, ExpressJS and Nginx Ingress

Question

I want to connect a React Native application using Socket.io to a server that is inside a Kubernetes Cluster hosted on Google Cloud Platform (GKE).

There seems to be an issue with the Nginx Ingress Controller declaration but I cannot find it.

I have tried adding nginx.org/websocket-services; rewriting my backend code so that it uses a separate NodeJS server (a simple HTTP server) on port 3004, then exposing it via the Ingress Controller under a different path than the one on port 3003; and multiple other suggestions from other SO questions and Github issues.

Information that might be useful:

1. Cluster master version: 1.15.11-gke.15
2. I use a Load Balancer managed with Helm (stable/nginx-ingress) with RBAC enabled
3. All deployments and services are within the namespace gitlab-managed-apps
4. The error I receive when trying to connect to socket.io is: Error: websocket error

For the front-end part, the code is as follows:

App.js

const socket = io('https://example.com/app-sockets/socketns', {
    reconnect: true,
    secure: true,
    transports: ['websocket', 'polling']
});

I expect the above to connect me to a socket.io namespace called socketdns.

The backend code is:

app.js

const express = require('express');
const app = express();
const server = require('http').createServer(app);
const io = require('socket.io')(server);
const redis = require('socket.io-redis');

io.set('transports', ['websocket', 'polling']);
io.adapter(redis({
    host: process.env.NODE_ENV === 'development' ? 'localhost' : 'redis-cluster-ip-service.gitlab-managed-apps.svc.cluster.local',
    port: 6379
}));
io.of('/').adapter.on('error', function(err) { console.log('Redis Adapter error! ', err); });

const nsp = io.of('/socketns');

nsp.on('connection', function(socket) {
    console.log('connected!');
});

server.listen(3003, () => {
    console.log('App listening to 3003');
});

The ingress service is:

ingress-service.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "7200"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "7200"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "7200"
    nginx.org/websocket-services: "app-sockets-cluster-ip-service"
  name: ingress-service
  namespace: gitlab-managed-apps
spec:
  tls:
  - hosts:
    - example.com
    secretName: letsencrypt-prod
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: app-cms-cluster-ip-service
          servicePort: 3000
        path: /?(.*)
      - backend:
          serviceName: app-users-cluster-ip-service
          servicePort: 3001
        path: /app-users/?(.*)
      - backend:
          serviceName: app-sockets-cluster-ip-service
          servicePort: 3003
        path: /app-sockets/?(.*)
      - backend:
          serviceName: app-sockets-cluster-ip-service
          servicePort: 3003
        path: /app-sockets/socketns/?(.*)

Answer 1

The solution is to remove the nginx.ingress.kubernetes.io/rewrite-target: /$1 annotation.

Here is a working configuration: (please note that apiVersion has changed since the question has been asked)

Ingress configuration

ingress-service.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "64m"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
  name: ingress-service
  namespace: default
spec:
  tls:
  - hosts:
    - example.com
    secretName: letsencrypt-prod
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: app-sockets-cluster-ip-service
            port:
              number: 3003
        path: /app-sockets/?(.*)
        pathType: Prefix

On the service (Express.js):

app.js

const redisAdapter = require('socket.io-redis');

const io = require('socket.io')(server, {
    path: `${ global.NODE_ENV === 'development' ? '' : '/app-sockets' }/sockets/`,
    cors: {
        origin: '*',
        methods: ['GET', 'POST'],
    },
});

io.adapter(redisAdapter({
    host: global.REDIS_HOST,
    port: 6379,
}));

io.of('/').adapter.on('error', err => console.log('Redis Adapter error! ', err));

io.on('connection', () => { 
//...
});

The global.NODE_ENV === 'development' ? '' : '/app-sockets' bit is related to an issue in development. If you change it here, you must also change it in the snippet below.

In development the service is under http://localhost:3003 (sockets endpoint is http://localhost:3003/sockets).

In production the service is under https://example.com/app-sockets (sockets endpoint is https://example.com/app-sockets/sockets).

On frontend

connectToWebsocketsService.js

/**
 * Connect to a websockets service
 * @param tokens {Object}
 * @param successCallback {Function}
 * @param failureCallback {Function}
 */
export const connectToWebsocketsService = (tokens, successCallback, failureCallback) => {
    //SOCKETS_URL = NODE_ENV === 'development' ? 'http://localhost:3003' : 'https://example.com/app-sockets'
    const socket = io(`${ SOCKETS_URL.replace('/app-sockets', '') }`, {
        path: `${ NODE_ENV === 'development' ? '' : '/app-sockets' }/sockets/`,
        reconnect: true,
        secure: true,
        transports: ['polling', 'websocket'], //required
        query: {
            // optional
        },
        auth: {
            ...generateAuthorizationHeaders(tokens), //optional
        },
    });

    socket.on('connect', successCallback(socket));
    socket.on('reconnect', successCallback(socket));
    socket.on('connect_error', failureCallback);
};

Note: I wasn't able to do it on the project mentioned in the question, but I have on another project which is hosted on EKS, not GKE. Feel free to confirm if this works for you on GKE as well.

Answer 2

Just change annotations to

nginx.ingress.kubernetes.io/websocket-services: "app-sockets-cluster-ip-service"

instead of

nginx.org/websocket-services: "app-sockets-cluster-ip-service" 

Mostly it will resolve your issue.