how to use async.parallelLimit to maximize the amount of (parallel) running processes?

Question

Is it possible to set a Limit to parallel running processes with async.parallelLimit ? I used following very simple code to test how it works.

var async = require("async");
var i = 0;

function write() {
    i++;
    console.log("Done", i);
}

async.parallelLimit([
    function(callback) {
        write();
        callback();
    }
], 10, function() {
    console.log("finish");
});

Of course everything what I got back was this:

Done 1
finish

In my case I have a function wich is called very often, but I only want it to run only 5 times simultaneously. The other calls should be queued. (Yes I know about async.queue but, this is not what I want in this question).

So now the question is, is this possible with async.parallelLimit?

//EDIT:

I actually have something similar to this:

for(var i = 0; i < somthing.length; i++) { //i > 100  for example
    write();
}

And 'cause of the non synchronicity of node.js this could run 100 times at the same time. But how shell I limit the parallel running processes in this case?

Answer 1

Very short answer; Yes. That's exactly what asyncParallelLimit does.

In your case, you are passing only one function to parallelLimit. That's why it only get's called once. If you were to pass an array with this same function many times, it will get executed as many times as you put it in the array.

Please note that your example function doesn't actually do any work asynchronously. As such, this example function will always get executed in series. If you have a function that does async work, for example a network request or file i/o, it will get executed in parallel.

A better example-function for a async workload would be:

function(callback) {
    setTimeout(function(){
        callback();
    }, 200);
}

Answer 2

For completion, to add to the existing answer, if you want to run the same function multiple times in parallel with a limit, here's how you do it:

  // run 'my_task' 100 times, with parallel limit of 10

  var my_task = function(callback) { ... };
  var when_done = function(err, results) { ... };

  // create an array of tasks
  var async_queue = Array(100).fill(my_task);

  async.parallelLimit(async_queue, 10, when_done);