NodeJS and Socket.io in production: Dealing with state

Question

In the development environment I have a single nodejs process which have socket.io listening on a given port. Once the client connects, an state needs to be maintained. The nodejs script do some work over files and sends the status of the processing through sockets. It's like a batch which may never end.

If the client closes the browser and then open again (in the development environment), the web page simply connects again to the socket and grabs the current state of the process, which will be handling the files processing in the background for that specific user.

In order to have a single process per user and ensure that the users always reconnect to that same process they started, on the same port, I need to manage those process, one per user as a service on the server and book the ports for each user.

How would I build this scenario as a single nodejs entry point that forks the processes and route the socket connections accordingly using perhaps Nginx, PM2 and or Nodejs clusters. In other words, which is the best architecture for production in a scenario like that?

About the state problem:

The state I'm holding are not simply variables, which can be stored in database. I have continuous file read streams that are processed one after another in the order the user has configured through the web page connected using sockets to that process. This process also connect to another socket server-to-server and the connection must be kept.

Answer 1

I would probably go with exposing the pids of your processes to your routing script and making use of sessions and cookies, saving them to a Redis DB along with the pid assigned to the user.

So on connection, you set a cookie and a session, maybe with a datetime-hash for the user ID (depending on whether you have user accounts associated with your clients), save it in the user cookie and save the pid in your session.

Then you can redirect your users to the corresponding fork of your process in your router.

Edit

I'd use a nodescript as the proxy. I found this answer, since you didn't post code I don't know how exactly you're creating your processes, so I'll use it for reference:

const socketio = require('socket.io');
const redis = require('redis');

on.connect((req, res) => {
    let uid,
        pid,
        port;

    // if user does not have the cookie yet
    if (!req.cookies.uid) {
        uid = getNewUserID(req); // create uid from information from the request, set a cookie later
        pid = createWorker(uid); // generate new socket.io worker in cluster and return pid
        port = getSocketPort(pid); // get socket port from worker

        // save acquired variables to database
        redis.set(uid, [pid, port], function(err, reply) {
            if (err) throw err;
            console.log(reply);
        });
    };

    uid = req.cookies.uid; // get uid from request
    pid = getPid(uid); // get pid from redis

    socketio(getSocketPort(pid)); // connect client to port of correct worker
});

for your readstreams, you can just do them inside the corresponding worker. 1 Worker equals 1 socket.io process

Answer 2

don`t save your state on the memory. save in a db(redis is a good db for that as it memory based and super fast) and then you dont have to worry if the socket reconnect to the same process or even to the same server (if you have multiple servers with load-balances ) .