Run tasks in multiple threads on node.js

Question

I need some help with node.js as i'm a newbie to it. I have an array of data and have to run functions with that data in multiple threads (using all CPU cores). In usual for me languages i do create threadpool with number of threads, push some tasks to it, wait until finish, send more tasks to queue. But i can't figure it out how to do that in nodejs.

PS: i'm using latest 0.8 branch.

I think you need this: [cluster](http://nodejs.org/api/cluster.html) — Rishi Diwan, Aug 19 '12 at 10:07
event-driven programming is pretty unusual for me now.. Need some help with examples of using with my task.. — elgato, Aug 19 '12 at 10:12
Since Node.js doesn't have threads, you need to think more in terms of traditional UNIX programming concepts (i.e. forking, spawning and IPC—Inter Process Communication). There's a NPM module called hook.io that makes the IPC nice and Node-like, rather than defining your own payload format etc, as would typically be the case: https://github.com/hookio/hook.io — d11wtq, Aug 19 '12 at 11:08
Just a note: a similar question with possibly useful answers can be found here: *How to create threads in nodejs* -> https://stackoverflow.com/questions/18613023/how-to-create-threads-in-nodejs/47631197 — Heinrich Ulbricht, Dec 04 '17 at 10:36

score 4 · Accepted Answer · answered Aug 19 '12 at 10:36

Nodejs is built to run on a single process, but you can spawn other processes. The cluster module uses the fork method from the child_process module, but it's intended for distributing connections of a server across processes and sharing the same port.

I think you want exec. Example:

var exec = require('child_process').exec,
child;

var array = ["a", "b", "c", "d", "e"]; // your array of data

var n = array.length; // length
var done = 0; // jobs done
var i = n; // iterator

while(i--) { // reverse while loops are faster
    (function (argument) { // A closure
        child = exec('node otherFunction.js '+argument, // Spawn the process, with an item from your array as an argument.
          function (error, stdout, stderr) {
            if (error === null) {
                done += 1;
                if (done === n) {
                    console.log('Everything is done');
                }
            }
        });
    })(array[i]);
}

The above is of course bad code and not even tested, but I think it would work. All you have to do is call the function you wanted to call on the array's items in otherFunction.js, inside that you would find the arguments in process.argv.

// reverse while loops are faster - Definitely worth trading readability for such huge performance gains. Only on SO — Mâtt Frëëman, Aug 19 '12 at 13:19
if you care about speed, do not create a closure inside a loop it makes you lose a lot (much more than looping with ++ instead of --) http://jsperf.com/closure-vs-name-function-in-a-loop/2 also: http://jslinterrors.com/dont-make-functions-within-a-loop — Jonathan Muller, Mar 10 '15 at 13:48
This thing is using `exec`, which spawns a new process. How do you think do the optimizations you mentioned compare to spawning a new process? I think the difference is more like a 1000 times... — Matthias Hryniszak, Mar 16 '22 at 22:13

Run tasks in multiple threads on node.js

1 Answers1