1

The websocket (using @stomp/stompjs library) connection works ok on local development bench. Running the app on Azure AKS platform using an NGINX proxy for serving the React app (with also NGINX ingress controller) also works fine. The challenge, however, is the websocket connection in the latter environment.

Firefox browser returns the following error when trying to connect:

Uncaught (in promise) DOMException: An invalid or illegal string was specified ...
    i stomp-handler.ts:31
    (Async: promise callback)
    s stomp-handler.ts:31
    da stomp-handler.ts:31
    da stomp-handler.ts:31
    value client.ts:404
    value client.ts:401

In React, the websocket configuration is:

const server = {
        name: process.env.REACT_APP_BASE_URL_SOCKET,
}
...
const client = new Client();
            client.configure({
                brokerURL: server.name,
    ...

The environment variable is:

REACT_APP_BASE_URL_SOCKET=/ws

NGINX configuration for React app is specified as:

...
http {
    ...
    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }

    server {
        # listen on port 80
        listen 80;
        
        server_name  foo.com;
                 
        gzip off;
        proxy_max_temp_file_size 0;

        # save logs here
        access_log /var/log/nginx/access.log compression;
        
        root  /usr/share/nginx/html;
        index  index.html index.htm;
        
        proxy_set_header HOST $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        
        location / {            
            try_files $uri $uri/ /index.html =404;
        }
        
        location /ws {
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "Upgrade";
            proxy_pass http://bar-api-service.default.svc.cluster.local:8000;
        }
        ...
    }
}

With Docker file as:

FROM nginx:alpine
COPY build/ /usr/share/nginx/html

# Copy our custom nginx config
COPY nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

ENTRYPOINT ["nginx", "-g", "daemon off;"]

Meanwhile, the Kubernetes NGINX Ingress controller is configured as:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/websocket-services: bar-ui-service
    nginx.org/websocket-services: bar-ui-service
spec:
  tls:
    - hosts:
      - foo.com
      secretName: tls-secret
  rules:
    - host: foo.com
      http:
        paths:
          - pathType: Prefix
            backend:
              service:
                name: bar-ui-service
                port:
                  number: 80
            path: /

With "bar-ui-service" as:

apiVersion: v1
kind: Service
metadata:                     
  name: bar-ui-service
spec:
  type: NodePort      
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
  selector:
    app: bar-ui

And "bar-api-service" as:

apiVersion: v1
kind: Service
metadata:
  name: bar-api-service
spec:
  selector:
    app: bar-api
    tier: backend
  ports:
    port: 8000
    targetPort: 8000

In the Spring Boot API serving the websocket, Spring Security is used, with OAuth2 resource server. Configuration of HttpSecurity is:

@Override
    protected void configure(HttpSecurity http) throws Exception {
        http.httpBasic().disable()
                .formLogin().disable()
                .and()
                .cors()
                .and()
                .authorizeRequests()
                .antMatchers("/ws").permitAll()
            

And websocket broker config as:

@Configuration
@EnableWebSocketMessageBroker
public class WebSocketBrokerConfig implements WebSocketMessageBrokerConfigurer {

    @Override
    public void configureMessageBroker(MessageBrokerRegistry registry) {
        registry.enableSimpleBroker("/queue", "/topic");
        registry.setApplicationDestinationPrefixes("/app");
    }

    @Override
    public void registerStompEndpoints(StompEndpointRegistry registry) {
        registry.addEndpoint("/ws")
                .setAllowedOrigins("foo.com");
    }
}

How do I get the websocket to work in k8s?

nkmuturi
  • 248
  • 3
  • 10
  • Try adding http uprade annotations to your nginx ingress controller proxy_set_header Upgrade $http_upgrade; proxy_http_version 1.1; proxy_set_header X-Forwarded-Host $http_host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header Host $host; proxy_set_header Connection "upgrade"; proxy_cache_bypass $http_upgrade; – Rakesh Gupta Oct 23 '22 at 00:10
  • Hi Rakesh. No, adding http upgrade annotations in the controller proxy does not work. Interestingly, the react-stomp library, which connects the websocket initially with http, works just fine in my kubernetes setup. However, I have not gotten the @stomp/stompjs library, which prompted my question, to work. Cheers for trying to help. – nkmuturi Oct 24 '22 at 03:40

1 Answers1

0

To solve this challenge in Kubernetes, I had to change the WebSocket library. Simply put, the '@stomp/stompjs' library (v6.1.2) cannot upgrade from http to ws. After trying out several libraries (yes, JavaScript WebSocket libraries seem to be a dime a dozen out there!), I settled on react-stomp-hooks (v2.1.0).

For those who might be curious about the fuller solution, read on.

In React (TS), the 'react-stomp-hook' library comes with a convenient Provider component for connecting to the server. This may be added to App.tsx like so:

...
import {
    StompSessionProvider,
} from "react-stomp-hooks";
import WebSocketSubscriptions from './shared/WebSocketSubscription';
...
const socketHeaders = {
    login: token.userId,
    passcode: <can be anything>
}

return (
        <React.Fragment>
            {token &&
                <StompSessionProvider
                    url={server.name}
                    connectHeaders={socketHeaders}>
                    <WebSocketSubscriptions />
                </StompSessionProvider>}
...

The url ('server.name') is from an environment variable defined as:

REACT_APP_BASE_URL_SOCKET=/ws

for Kubernetes deployment, and:

REACT_APP_BASE_URL_SOCKET=http://localhost:8000/ws

in the local development environment.

In order to work, the web socket's connection header requires to pass down a user identifier to match with Spring Security's user authentication (shown below). In my example, this is provided by the OAuth 2.0 JWT token. However, the "passcode", although a required parameter, can be anything. Note the Provider enables subscriptions to be defined in a separate component (or components) using a useSubscription hook, for example:

import {
    useSubscription,
} from "react-stomp-hooks";

const WebSocketSubscriptions: React.FC = () => {

    useSubscription("/app/subscribe", (message) => {
        console.log(`..subscribed ${message}`);
    });
...

The React-serving NGINX proxy configuration is as follows (you can see the fuller implementation in my question above):

...
location /ws {
            proxy_pass http://bar-api-service.default.svc.cluster.local:8000;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_cache_bypass $http_upgrade;
        }
...

Meanwhile, the Kubernetes NGINX ingress controller need only deal with secure termination of traffic, like:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: foo-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    cert-manager.io/cluster-issuer: letsencrypt
spec:
  tls:
    - hosts:
      - foo.com
      secretName: tls-secret
  rules:
    - host: foo.com
    ...

The heavy-lifting now comes in the Spring Boot (v2.7.3) API where the WebSocket server resides. Addition of Spring Security is what makes the solution hairy.

At least the following dependencies are required:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-oauth2-resource-server</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-websocket</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.security</groupId>
    <artifactId>spring-security-messaging</artifactId>
    <version>5.6.2</version>
</dependency>

First let's look at the websocket's support files:

example websocket package

The AuthChannelInterceptorAdaptor class authenticates the user:

@Component
public class AuthChannelInterceptorAdaptor implements ChannelInterceptor {

    private static final String USERNAME_HEADER = "login";
    private static final String PASSWORD_HEADER = "passcode";
    private final WebSocketAuthenticatorService webSocketAuthenticatorService;

    @Autowired
    public AuthChannelInterceptorAdaptor(final WebSocketAuthenticatorService webSocketAuthenticatorService) {
        this.webSocketAuthenticatorService = webSocketAuthenticatorService;
    }

    @Override
    public Message<?> preSend(final Message<?> message, final MessageChannel channel)
            throws AuthenticationException {
        final StompHeaderAccessor accessor = MessageHeaderAccessor.getAccessor(message, StompHeaderAccessor.class);

        if (StompCommand.CONNECT == accessor.getCommand()) {
            final String username = accessor.getFirstNativeHeader(USERNAME_HEADER);
            final String password = accessor.getFirstNativeHeader(PASSWORD_HEADER);

            final UsernamePasswordAuthenticationToken user =
                    webSocketAuthenticatorService.getAuthenticatedOrFail(username, password);

            accessor.setUser(user);
        }
        return message;
    }
}

As mentioned earlier, for an API server using the OAuth 2.0 Resource library, only the username need be provided; password can be anything. Spring Boot Security simply authenticates the user id (from the websocket's header login) and accepts the connection.

The authentication is carried out in the WebSocketAuthenticatorService class:

/*
courtesy: Anthony Raymond: https://stackoverflow.com/questions/45405332/websocket-authentication-and-authorization-in-spring
 */
@Component
public class WebSocketAuthenticatorService {

    public UsernamePasswordAuthenticationToken getAuthenticatedOrFail(final String  username,
                                                                      final String password)
            throws AuthenticationException {

        if (username == null || username.trim().isEmpty()) {
            throw new AuthenticationCredentialsNotFoundException("Username was null or empty.");
        }
        if (password == null || password.trim().isEmpty()) {
            throw new AuthenticationCredentialsNotFoundException("Password was null or empty.");
        }
        // You can add your own logic for retrieving user from, say, db
        //if (fetchUserFromDb(username, password) == null) {
        //  throw new BadCredentialsException("Bad credentials for user " + username);

        // Null credentials, we do not pass the password along
        return new UsernamePasswordAuthenticationToken(
                username,
                null,
                Collections.singleton((GrantedAuthority) () -> "USER") // MUST provide at least one role
        );
    }
}

The WebSocketAuthorizationSecurityConfig class extends AbstractSecurityWebSocketMessageBrokerConfigurer, allowing configuration of inbound messages:

@Configuration
public class WebSocketAuthorizationSecurityConfig extends AbstractSecurityWebSocketMessageBrokerConfigurer {

    @Override
    protected void configureInbound(final MessageSecurityMetadataSourceRegistry messages) {
        // You can customize your authorization mapping here.
        messages.anyMessage().authenticated();
    }
    
    @Override
    protected boolean sameOriginDisabled() {
        return false;
    }

}

Then we come to the WebSocketBrokerConfig class, which implements WebSocketMessageBrokerConfigurer:

@Configuration
@EnableWebSocketMessageBroker
public class WebSocketBrokerConfig implements WebSocketMessageBrokerConfigurer {

    @Override
    public void configureMessageBroker(MessageBrokerRegistry registry) {
        registry.enableSimpleBroker("/queue", "/topic");
        registry.setApplicationDestinationPrefixes("/app");
    }

    @Override
    public void registerStompEndpoints(StompEndpointRegistry registry) {
        registry.addEndpoint("/ws")
                .setAllowedOrigins("http://localhost:3000", "foo.com")
                        .withSockJS();
    }
}

This in-memory broker is good for training and POCs. In production, however, a more robust provider like RabbitMQ or Kafka is called for.

Another implementation of the WebSocketMessageBrokerConfigurer is required for configuring the inbound channel. This is implemented in the class WebSocketSecurityConfig:

@Configuration
@Order(Ordered.HIGHEST_PRECEDENCE + 99)
public class WebSocketSecurityConfig implements WebSocketMessageBrokerConfigurer {

    @Autowired
    private AuthChannelInterceptorAdaptor authChannelInterceptorAdapter;

    @Override
    public void registerStompEndpoints(final StompEndpointRegistry registry) {
        // Endpoints are already registered on WebSocketConfig, no need to add more.
    }

    @Override
    public void configureClientInboundChannel(final ChannelRegistration registration) {
        registration.setInterceptors(authChannelInterceptorAdapter);
    }
}

Note the @Order annotation, which configures a high precedence for the channel.

The React app client move from a ws-based websocket connection to an http-brokered one also requires a CORS filter to work. The example filter looks like:

/*
courtesy of https://stackoverflow.com/users/3669624/cнŝdk
*/

@Component
public class CORSFilter implements Filter {

    private final List<String> allowedOrigins = Arrays.asList(
            "http://localhost:3000", "foo.com");

    public void destroy() {}

    public void doFilter(ServletRequest req, ServletResponse res, FilterChain chain) throws IOException, ServletException {
        // Lets make sure that we are working with HTTP (that is, against HttpServletRequest and HttpServletResponse objects)
        if (req instanceof HttpServletRequest && res instanceof HttpServletResponse) {
            HttpServletRequest request = (HttpServletRequest) req;
            HttpServletResponse response = (HttpServletResponse) res;
            
            // Access-Control-Allow-Origin
            String origin = request.getHeader("Origin");
            response.setHeader("Access-Control-Allow-Origin", allowedOrigins.contains(origin) ? origin : "");
            response.setHeader("Vary", "Origin");

            // Access-Control-Max-Age
            response.setHeader("Access-Control-Max-Age", "3600");

            // Access-Control-Allow-Credentials
            response.setHeader("Access-Control-Allow-Credentials", "true");

            // Access-Control-Allow-Methods
            response.setHeader("Access-Control-Allow-Methods", "POST, GET, OPTIONS, DELETE");

            // Access-Control-Allow-Headers
            response.setHeader("Access-Control-Allow-Headers",
                    "Origin, X-Requested-With, Content-Type, Accept, " + "X-CSRF-TOKEN");
        }

        chain.doFilter(req, res);
    }

    public void init(FilterConfig filterConfig) {
    }
}

The WebSocketController class simply provides an endpoint for the websocket's subscriptions and support for discrete '/queues'.

@Controller
public class WebSocketController {

    @MessageMapping("/subscribe")
    @SendToUser("/queue/notification")
    public String replyToAccountFromClient(@Payload String message,
                                           Principal user) {
        return String.format("hello: %s", message);
    }

    @MessageExceptionHandler
    @SendTo("/queue/errors")
    public String handleException(Throwable exception) {
        return exception.getMessage();
    }
}

Finally, the HttpSecurity configuration looks like the following:

@Override
    protected void configure(HttpSecurity http) throws Exception {
        http.httpBasic().disable()
                .formLogin().disable()
                .addFilterAt(corsFilter, BasicAuthenticationFilter.class)
                .csrf().ignoringAntMatchers(API_URL_PREFIX)
                .and()
                .cors()
                .and()
                .authorizeRequests()
                .antMatchers("/ws/**").permitAll()
                ...
                .oauth2ResourceServer(oauth2ResourceServer ->
                        oauth2ResourceServer.jwt(jwt ->
                                jwt.jwtAuthenticationConverter(
                                        getJwtAuthenticationConverter())))
                .sessionManagement().sessionCreationPolicy(SessionCreationPolicy.STATELESS);
}
nkmuturi
  • 248
  • 3
  • 10