0

I'm writing a WebSocket client in Node.js, and I want to perform asynchronous operations with the message data (i.e. storing in database), but also ensure that messages are operated on one at a time, in the order in which they were received.

To achieve this, I'm pushing messages on to an array in my onMessage handler, and simultaneously shifting messages off the array one at a time using a function that calls itself. Something like this:

const flushQueue = async () => {

  const nextMessage = messageQueue.shift();

  await doSomethingAsync(nextMessage); // i.e. store in db, etc.

  flushQueue();
}

Setting aside the issue of whether or not my system can keep up, this method works well, except the script eventually runs out of heap memory. A simple test reveals that shifting an element off an array actually increases the amount of heap memory used, whereas my assumption was that it reduced the memory used:

const a = [1,2,3,4,5,6];

a.shift();
a.shift();

console.log(process.memoryUsage());

Why does removing an element from an array increase the amount of memory used by the process? How can I store my set of queued WebSocket messages in a memory-efficient way, indefinitely?

oliakaoil
  • 1,615
  • 2
  • 15
  • 36

1 Answers1

1

I am almost sure the problem is in your tail call, and not shifting the array. Node and Javascript does not have proper tail call optimization. Some stats and post on TCO.

Long running loop without tail call optimization will eventually run out of memory.

I would drop this custom queue, and write directly to a database.

SQLite + better-sqlite3 driver is a good place to start.

// pseudo-code example (not tested)
ws.on('message', function incoming (data) {
  const stmt = db.prepare('INSERT INTO list(message) VALUES(?)')
  const result = stmt.run(data) // synchronous in-process write
  // do something with result or something else
});
Anastazy
  • 4,624
  • 2
  • 13
  • 22