0

Background

I have a NodeJS/Express application that uses WebSockets for certain endpoints that need to hold a connection for backend functions that take several minutes to complete.

An example of one of these functions is the creation and processing of an Excel document that is sent back to the client for download.

Problem

The creation of this Excel document is triggered by a button click on the frontend. After the button is clicked, a WebSocket message is sent to the server that identifies an Excel document needs to be created. A function is called that begins creating the document. As part of this document's creation, a number of API and function calls are made, which causes the whole process to take several minutes to complete. After 2 minutes of waiting for the document to be created and sent back, the server restarts, preventing the document from being created and sent to the client.

Important Notes

When I test my code locally, everything works as intended. Furthermore, I have a heartbeat (ping/pong) setup so the WebSocket connection isn't closed from inactivity. After the WebSocket connection is made, the connection stays active indefinitely; however, clicking the button that causes the Excel document to be created is what causes the server to restart after 2 minutes.

Code Snippets

Code for setting up the application, routes, WebSocket connections, etc.

const express = require('express');
const http = require('http');
const WebSocket = require('ws');

const app = express();
const server = http.createServer(app);
const wss = new WebSocket.Server({ server:server });

// More Application Setup Here...

wss.on('connection', function connection(ws) {
    ws.on('message', async function incoming(message) {
        if (message.toString() == '__ping__') {
            ws.send('__pong__');
            return;
        }

        let msg = JSON.parse(message);
        switch(msg.call) {
            case "create_excel":
                var workbook = await create_excel();
                ws.send(workbook); // Workbook is converted to buffer in real code
                break;
            default:
                break;
        }
    });
});

// Routes Here...

const port = process.env.PORT || 3000;
server.listen(port, () => console.log(`Listening on port ${port}...`));

Client code for connecting to socket and sending request for file:

const socket = new WebSocket("wss://myurl.com/path");

socket.onopen = function() {
    setInterval(ping, 30000);
};

socket.onmessage = function(event) {
    if (event.data == "__pong__") {
        pong();
        return;
    }

    if (typeof event.data == 'object') {
        # Call function to save file
        return;
    }

    # Handle other messages here...
};

function ping() {
    socket.send('__ping__');
    tm = setTimeout(5000);
}

function pong() {
    clearTimeout(tm);
}

button.onclick = function() {
    socket.send(JSON.stringify({ call: "create_excel" }));
}

Attempted Fixes

Prior to using WebSockets, I just used HTTP requests for this same function; however, that would fail after 2 minutes as well in the form of a 504 Gateway Error. I transitioned to WebSockets hoping to fix this issue. I've tried the solutions suggested here with no luck.

Could this be a server configuration problem? Are there other options for trying to increase the timeout?

GriffsAccount
  • 95
  • 1
  • 12
  • What kind of hosting are you using? This seems like it could be a hosting plan limitation that thinks it detects a "stuck" process and shuts you down. Also, are you 100% sure that long lived webSocket connections are supported in your hosting plan? Some shared hosting plans don't allow that. – jfriend00 Dec 29 '21 at 18:15
  • Do you get any info in any log, either your server log or a log associated with your hosting account on why it's being shut down? – jfriend00 Dec 29 '21 at 18:16
  • My code is deployed to VMware's Pivital Container Service (PKS); however, I unfortunately don't have access to any of the configuration/logs for the service. Which I suppose makes this issue hard to diagnose. I don't think support of long lived WebSocket connections is the problem here, as I get the same timeout when using HTTP requests instead of a WebSocket connection. – GriffsAccount Dec 29 '21 at 18:23
  • Something else to note. If I simulate the time it takes for the document to be created and processed by called a sleep function that waits over 2 minutes rather than the function to create the document, the websocket connection doesn't close and the server doesn't restart. – GriffsAccount Dec 29 '21 at 18:50
  • You need to get access to those logs. If the hosting provider is shutting you down for some reason (such as exceeding some allowable resource limit), then the reason should be in a log somewhere. If it's something your own code is doing and causing a silent crash, then we'd need to see the code to have any chance at helping and we'd probably look for places where you are missing proper error handling and logging of problems. – jfriend00 Dec 29 '21 at 18:54
  • If the code runs as intended locally, no matter how long it takes to process to document, it shouldn't be the code causing the problem, correct? – GriffsAccount Dec 29 '21 at 19:00
  • 1
    Not necessarily. You could have hosting resource limits you're hitting. You could have installation issues with some modules you depend on. The code could have issues on the platform/OS the hosting provider is using. The hosting could have restrictions on some things your code is doing. If it were me, I'd get access to the hosted logs and I'd instrument the heck out of the code so I can figure out where it is in my code when it shuts down. Beyond that, I don't really have anything else to contribute here. – jfriend00 Dec 29 '21 at 19:11
  • I appreciate the suggestions. Will do my best to get access to the logs and go from there. Thanks! – GriffsAccount Dec 29 '21 at 19:25

0 Answers0