1

My goal is to create a parent process that streams messages to the child processes indefinitely. To test this, i tried the below code for 150M messages to a child process that does nothing.

  1. (Possible Answer Edited Below) Why would this code run out of memory?
  2. I am using the ofe module but I do not know what to look for in the resulting heapdump, what method could I have used to search the heapdump for clues?

code:

server.js

//  After 20M messages i get: 
//  FATAL ERROR: CALL_AND_RETRY_2 Allocation failed - process out of memory


var cp = require('child_process');
var xrange = require('xrange');
var maxChildren = require('os').cpus().length;
var util = require('util');
require('ofe').call();

var childError = function(err) {
    console.log('Error: ' + err);
};

var childExit = function(code, signal) {
    console.log('Exit Code: ' + code);
    console.log('Exit Signal: ' + signal);
};

var childClose = function(code, signal) {
    console.log('Close Code: ' + code);
    console.log('Close Signal: ' + signal);
};

var childDisconnect = function() {
    console.log('Disconnect');
};

var childMessage = function(msg, handle) {
    console.log('Msg: ' + msg);
};

var createChild = function(){
    var child = cp.fork("./child");
    child.on('error', childError);
    child.on('exit', childExit);
    child.on('close', childClose);
    child.on('disconnect', childDisconnect);
    child.on('message', childMessage);
    console.log("Child Created: " + child.pid);
    return child;
}

var createChildren = function(){
    var children = [];
    xrange(maxChildren).each(function(i) {
        children[i] = createChild();
    });
    return children;
}

var sendMessages = function(children) {
xrange(150000000).each(function(num) {
var idx = num % maxChildren;
if (num % 1000000 == 0) {
  console.log(num);
}
children[idx].send(num);
});
};

child.js (contents)

process.on('message', function(msg) {});

Answer

Since this code is asynchronous , the parent process will send all the messages to the children without waiting for them to be processed, which seems to overload the clients. I believe the solution is to send a message back from the clients to 'pull' the next number after processing. Assuming this is the answer, I have a follow up question.

1) I'd like to write this but i'm not sure how to turn the xrange into a generator without the ability to yield (no-harmony nodejs), is there a callback solution? :

function getNextNumber(){
  //pull the next available number from the 150M
}

child.on('message',function(msg) {
  child.send(getNextNumber());
});

Followup

If I just wanted to iterate, this seems to be the way: https://stackoverflow.com/a/5784473/1578888 .

If I am reading this alternate answer correctly, it seems that implementing the xrange call as a true generator is not possible pre-harmony: https://stackoverflow.com/a/7442013/1578888

This is the code i ended up using (also renamed server.js to parent.js):

parent.js

var cp = require('child_process');
var xrange = require('xrange');
var maxChildren = require('os').cpus().length;
var util = require('util');
require('ofe').call();

var childError = function(err) {
    console.log('Error: ' + err);
};

var childExit = function(code, signal) {
    console.log('Exit Code: ' + code);
    console.log('Exit Signal: ' + signal);
};

var childClose = function(code, signal) {
    console.log('Close Code: ' + code);
    console.log('Close Signal: ' + signal);
};

var childDisconnect = function() {
    console.log('Disconnect');
};

var childMessage = function(msg, handle) {
        //no output anymore!
    //console.log('Msg: ' + msg);
};

var createChild = function(){
    var child = cp.fork("./child.js");
    child.on('error', childError);
    child.on('exit', childExit);
    child.on('close', childClose);
    child.on('disconnect', childDisconnect);
    child.on('message', childMessage);
    console.log("Child Created: " + child.pid);
    return child;
}
    var getNextFn = (function () {
        var i = 0
        return function(cb) {
            i = i + 1;
            if (i < 150000000) {
                return i;
            } else {
                return null;
            }
        }
    })();

var createChildren = function(){
    var children = [];
    xrange(maxChildren).each(function(i) {
        var child = createChild();
            child.on('message', function(msg) {
                var next = getNextFn();
                if (next) {
                    if (next % 1000000 == 0) {
                        console.log(next + " " + new Date().toISOString());
                    }   
                    child.send(next);
                } else {
                    child.kill();
                }
            });
            children[i] = child;
    });
    return children;
}

var c = createChildren();

child.js

process.on('message', function(msg) {   
process.send({});
});

process.send({});
Community
  • 1
  • 1
  • How much memory does your system have free, before running this? – Dancrumb Dec 31 '13 at 15:22
  • @Dancrumb 4.25 GB free of 8.00 GB on osx – functionalcoder Dec 31 '13 at 15:27
  • does it work with fewer messages? do you really need it to handle 150M messages at once? – Plato Dec 31 '13 at 16:49
  • Yes it works up to 20M, I do not need them to be processed at the same time. Maybe that is the problem, maybe I should switch to an approach where the children msg the process when they need an item... how would i do that without generators for the 150M items? – functionalcoder Dec 31 '13 at 16:53
  • Looks like you can simply call process.send from within the child: http://stackoverflow.com/questions/10394206/how-to-send-message-to-parent-process – Plato Dec 31 '13 at 16:57
  • Edited OP to reflect possible answer and followup question – functionalcoder Dec 31 '13 at 17:09
  • @functionalcoder I'm pretty sure that is the problem. Try `async` npm module's `timesSeries` function - https://github.com/caolan/async#timesSeries – Dogbert Dec 31 '13 at 17:18
  • Agreed. That is way too much synchronous queuing. You should probably be sending the message and ACKing a response from the child so you know it has received it and that you can send another. – loganfsmyth Dec 31 '13 at 17:24
  • Thanks everyone, i updated the answer with the code I ended up with. – functionalcoder Dec 31 '13 at 18:52

0 Answers0