running async code in parallel

Question

Using the aync library, is it possible to create truly asynchronous tasks without using pre-built asynchronous i/o functions?

For example, if you run this code, it will always run the functions sequentially, since node.js is "single-threaded".

var async = require('async');

async.parallel([

    function(){
        console.log('1')
    },

    function(){
        console.log('2')
    },

    function(){
        console.log('3')
    },

    function(){
        console.log('4')
    }

]);

We aren't supposed to block Node - yes - but node.js runs on a single-core. If we wanted to utilize other cores, we could spawn processes and use the async library to good effect in that way.

Is there a best practice way to spawn async processes on other cores, besides the cluster module?

This is probably a better question for [programmers.stackexchange.com](http://programmers.stackexchange.com/). — Sumner Evans, May 28 '15 at 01:05
This question is on-topic here but probably needs some editing. Also, "best practice" questions tend to fare poorly at Programmers.SE. Please read: **[What goes on Programmers.SE? A guide for Stack Overflow](http://meta.programmers.stackexchange.com/q/7182/22815)**. — , May 28 '15 at 01:08
Node is not multithreaded, but it is concurrrent. Parallel is not necessarily running things sequentially. The results come back in order, but the operations may run in any order. — elclanrs, May 28 '15 at 01:08
@Snowman, where should "best practice" questions like this go? Is SE not a good place to ask that sort of question? — Sumner Evans, May 28 '15 at 01:11
@jsve the problem with "what is the best practice?" is it is typically an opinion poll. The fact that there are multiple options from which to choose the "best" indicates that there either is no best way or it is context-dependent. You are better off saying "here is my problem, here is what I tried, how do I accomplish X?" — , May 28 '15 at 01:17
@AlexMills, I recommend finding a programmer that you respect and ask them. If you don't know any that are experts on this, you might want to ask this question on one of the chat boards. — Sumner Evans, May 28 '15 at 01:18
`but node.js runs on a single-core` what makes you think this? — Marc, May 28 '15 at 02:09
@jsve: One indirect way of asking about best practice is to post your code for review/critique on http://codereview.stackexchange.com/ — slebetman, May 28 '15 at 03:11
http://stackoverflow.com/questions/2387724/node-js-on-multi-core-machines — Alexander Mills, May 28 '15 at 03:11
@Marc: That's what single-threaded means. What makes you think node is not single-threaded (with the exception of the disk I/O thread)? — slebetman, May 28 '15 at 03:15

score 2 · Answer 1 · answered May 28 '15 at 09:13

2

You can use something like - https://github.com/rvagg/node-worker-farm to run your tasks inside child processes.

answered May 28 '15 at 09:13

zag2art

4,869
1
29
39

Alexander Mills · Accepted Answer · 2017-08-28T22:21:58.877

Ok, I figured out a good solution. For the simplest possible situation, we use the core module 'child_process'. You don't even have to run npm install to use this module, it is a core module.

To solve this one, I used information from here:

http://blog.carbonfive.com/2014/02/28/taking-advantage-of-multi-processor-environments-in-node-js/

https://nodejs.org/api/child_process.html

//main.js

var cp = require('child_process');
var async = require('async');

async.parallel([

    function one(callback){
        console.log('1');
        var k = cp.fork('./doOther.js',['01']);
        k.once('message',function(msg){
            console.log(msg);
            callback();
        });
    },
    function two(callback){
        console.log('2');
        var k = cp.fork('./doOther.js',['02']);
        k.once('message',function(msg){
            console.log(msg);
            callback();
        });
    },
    function three(callback){
        console.log('3');
        var k = cp.fork('./doOther.js',['03']);
        k.once('message',function(msg){
            console.log(msg);
            callback();
        });

    },
    function four(callback){
        console.log('4');
        var k = cp.fork('./doOther.js',['04']);
        k.once('message',function(msg){
            console.log(msg);
            callback();
        });

    }

],function(){

    console.log('done.');
    process.exit(0);

});

a separate file in the same directory:

//doOther.js

process.on('message', function(msg) {
    console.log('child process got message:', msg);
});

setTimeout(function(){
    process.send({ foo: process.argv[2] });
},1000);

put the two files in the same directory, and away you go. Feel free to ask any questions. I will respond.

The above could be simplified to something like this:

  async.parallel([1,2,3,4].map(function(val){

       return function(cb){
            console.log(val);
            var k = cp.fork('./other-script.js',[val]);
            k.once('message',function(msg){
                console.log(msg);
                cb(null, msg);
            });
        },

     }),
     function(err, results){

        console.log('done.');
        process.exit(0);

    });

Yep the above is verbose, and you can probably use Array.prototype.map to avoid having to declare each separate function, since there is so much code sharing. — Alexander Mills, Aug 28 '17 at 22:17

running async code in parallel

2 Answers2