NodeJS Event Emitter Blocking Issue

Question

I have a node application handling some ZeroMQ events coming from another application utilizing the Node-ZMQ bindings found here: https://github.com/JustinTulloss/zeromq.node

The issue I am running into is one of the operations from an event takes a long time to process and this appears to be blocking any other event from being processed during this time. Although the application is not currently clustered, doing so would only afford a few more threads and doesn't really solve the issue. I am wondering if there is a way of allowing for these async calls to not block other incoming requests while they process, and how I might go about implementing them.

Here is a highly condensed/contrived code example of what I am doing currently:

var zmq = require('zmq');
var zmqResponder = zmq.socket('rep');
var Client = require('node-rest-client').Client;
var client = new Client();

zmqResponder.on('message', function (msg, data) {
  var parsed = JSON.parse(msg);
  logging.info('ZMQ Request received: ' + parsed.event);
  switch (parsed.event) {
    case 'create':
        //Typically short running process, not an issue
    case 'update':
        //Long running process this is the issue
        serverRequest().then(function(response){
            zmqResponder.send(JSON.stringify(response));
        });
     }

});

function serverRequest(){
    var deferred = Q.defer();
      client.get(function (data, response) {
      if (response.statusCode !== 200) {
        deferred.reject(data.data);
      } else {
        deferred.resolve(data.data);
      }
  });
    return deferred.promise;
}

EDIT** Here's a gist of the code: https://gist.github.com/battlecow/cd0c2233e9f197ec0049

Answer 1

I think, through the comment thread, I've identified your issue. REQ/REP has a strict synchronous message order guarantee... You must receive-send-receive-send-etc. REQ must start with send and REP must start with receive. So, you're only processing one message at a time because the socket types you've chosen enforce that.

If you were using a different, non-event-driven language, you'd likely get an error telling you what you'd done wrong when you tried to send or receive twice in a row, but node lets you do it and just queues the subsequent messages until it's their turn in the message order.

You want to change REQ/REP to DEALER/ROUTER and it'll work the way you expect. You'll have to change your logic slightly for the ROUTER socket to get it to send appropriately, but everything else should work the same.

---

Rough example code, using the relevant portions of the posted gist:

var zmqResponder = zmq.socket('router');

zmqResponder.on('message', function (msg, data) {
    var peer_id = msg[0];
    var parsed = JSON.parse(msg[1]);
    switch (parsed.event) {
        case 'create':
            // build parsedResponse, then...
            zmqResponder.send([peer_id, JSON.stringify(parsedResponse)]);
            break;
    }
});

zmqResponder.bind('tcp://*:5668', function (err) {
    if (err) {
        logging.error(err);
    } else {
        logging.info("ZMQ awaiting orders on port 5668");
    }
});

... you need to grab the peer_id (or whatever you want to call it, in ZMQ nomenclature it's the socket ID of the socket you're sending from, think of it as an "address" of sorts) from the first frame of the message you receive, and then use send it as the first frame of the message you send back.

By the way, I just noticed in your gist you are both connect()-ing and bind()-ing on the same socket (zmq.js lines 52 & 143, respectively). Don't do that. Inferring from other clues, you just want to bind() on this side of the process.