46

I have three applications talking to each other. A websocket server (1) that accepts connections from browsers, parses url to see what data is required, serves it to client if it has the data in memory, if not asks for it from another application called "fetcher" (2). Fetcher receives this job, requests it from a simple API (3) that returns JSON data, and sends it back to websocker server which publishes it to connected clients. "Fetcher" then starts to periodically check if there are updates to that url/job, and sends the new data to websocket server as they occur.

I use socket.io for client-websocket server communication. Websocket server and fetcher talks via ZMQ sockets.

I load test websocket server with 130 connections. Websocket server publishes 160KB of data to 130 clients every second. In the beginning, it uses 170mb of RAM for 130 connections, but swiftly this increases to 1GB although there are no new connections. Then socket.io's heartbeat signals starts to fail, resulting in dropped connections.

I use Nodetime to take heap snapshots. Right after 130th client connects, here's how memory looks:

enter image description here

346 Buffer objects with total 44MB.

In four minutes, number of Buffer objects increases dramatically (again, without new connections): there are 3012 of them with a total memory of 486MB. After 10 more minutes, there are 3535 of them with a total memory consumpion of 573MB.

I used Mozilla's memwatch to find out which line adds to memory, and found to be this function:

function notifyObservers(resourceId) {
  var data = resourceData[resourceId];
  io.sockets.in(resourceId).emit('data', data);
}

If I comment these lines out, memory usage stays the same so that's another confirmation.

Any ideas how this can happen? I call this function inside ZMQ's subscriber socket method and I suspect it has something to do with that.. This is the resulting code if I remove functions and merge them into one:

// Receive new resource data
const resourceUpdatedSubscriber = zmq.socket('sub').connect('tcp://localhost:5433');
resourceUpdatedSubscriber.subscribe('');

resourceUpdatedSubscriber.on('message', function (data) {
  var resource = JSON.parse(data);

  resourceData[resource.id] = resource.data;

  io.sockets.in(resourceId).emit('data', resourceData[resource.id]);
});

All my code (including the load test) is public and you can find this web socket server here: https://github.com/denizozger/node-socketio/blob/master/server.js See line 138.

I started learning Javascript and Node.js two months ago so any comments are welcome, thanks in advance!

Taylan Aydinli
  • 4,333
  • 15
  • 39
  • 33
Deniz Ozger
  • 2,565
  • 23
  • 27
  • 2
    this might not be the final solution but a good pointers for further investigation for sure: 1) http://www.tuicool.com/articles/yUR3q2 2) https://github.com/jmatthewsr-ms/node-slab-memory-issues – shoen Dec 13 '13 at 15:08
  • Thanks a lot Dawid, indeed those don't appear to be final solution but they inspired me. I replicated again with these simple lines: `resourceUpdatedSubscriber.on('message', function (data) { io.sockets.emit('data', data); });` The important point is: if I serve the same data statically (ie. `var bigData = require('./bigdata.json');`, I don't have any issues. Whenever I use the data coming from ZMQ publisher and 'emit' it, then I have the memory issue. Still working on it... – Deniz Ozger Dec 13 '13 at 16:33
  • 1
    Quick update, serving static json only delays the problem. At 230 observers memory leak is apparent again.. – Deniz Ozger Dec 13 '13 at 17:33
  • 2
    I re-implemented the application with Engine.IO (https://github.com/denizozger/node-engine.io-server). The issue still exists but this time with above 210 users (35MB data transfer / second). – Deniz Ozger Dec 18 '13 at 15:58
  • Any update on this? Node has plugged a few memory leaks since December, have you had any improvement, or did you identify and fix any specific issue? – Jason May 28 '14 at 20:02
  • @Jason No not really. Having heard that there were some related amendments in Node recently, I'm keen on trying to replicate and see if the issue still persists in the following weeks. I'll update here. – Deniz Ozger May 29 '14 at 10:46
  • why is resourceUpdatedSubscriber const? – Jacek Pietal Jun 04 '14 at 20:42
  • this is from 2013 probably no longer relevant – fmsf Jul 05 '14 at 14:26
  • How did you find the exact line of problem with Mozilla's node-memwatch ? – Deyan Vitanov Dec 27 '14 at 09:42

2 Answers2

2

NodeJs may be use Windows Socket API ( which include memory leaks , old known bug ) https://connect.microsoft.com/VisualStudio/feedback/details/605621/winsock-c-program-causes-memory-leak

The problem is the WSACleanup will never be called until you shutdown the network services. ( Mixing up ZMq or Nodejs won't change that fact )

Now, over the time, you will lock more pages of memory and such notice an increasing after the first memory page is full.

danbo
  • 147
  • 7
0

maybe try adding

var resourceData;

somewhere in your code because maybe your memory leak has to do with globals

read more about globals here: https://gist.github.com/hallettj/64478

Jacek Pietal
  • 1,980
  • 1
  • 18
  • 27
  • Not really sure if your answer was THE answer or not, but the referenced gist is a great read. – HeadCode Oct 11 '14 at 16:04
  • resouceData is actually defined already at that line: https://github.com/denizozger/node-socketio/blob/master/server.js#L54 – Deniz Ozger Dec 10 '14 at 18:18