I really like the Async Library:
https://github.com/caolan/async
which gives you a whole bunch of options of running sequential, asynchronous functions.
async.each([..], function(callback){
// Iterator function
callback();
});
Works in the same way as jQuery's each, but unlike jQuery, you get more control, such as:
async.eachSeries([...], function(callback){
// Will handle each element one at a time
callback();
});
And async.eachLimit, which means you cancontrol the number of tasks being run at any given time -so that x number of tasks are being run in parallel at any given time;
So, for example:
async.eachLimit([...], 2, function(callback){
// This will run the tasks simultaneously
callback();
});
Will run the iterator functions over all the elements in the array, but will limit the number of concurrently running tasks to 2. If you want 4 tasks, just change the second argument to 4, etc...
So, for example:
async.eachLimit([...], 2, function(callback){
// This will run up to 2 tasks simulatenously
callback(); // If successfull
});
If you need a timeout to make the operation fail silently after a timeout (jobs that pass the time out just don't get finished, and the rest of the queue moves on.
async.eachLimit([...], 2, function(callback){
// This will run up to 2 tasks simulatenously
callback(); // If successfull
setTimeout(function(){
return callback();
}, timeoutLength);
});
If you need a timeout to make the operation fail un-silently, (if there's one error, everything will stop).
async.eachLimit([...], 2, function(callback){
// This will run up to 2 tasks simulatenously
callback(); // If successfull
setTimeout(function(){
return callback("Error");
}, timeoutLength);
});
I'm not sure what the exact requirements of your job are, but I think the async library is a good candidate for this kind of stuff, and has a lot of control flow flexibility to get your job done.