1

Lets take an example where I have a huge array with elements being stringified JSON. I want to iterate over this array and convert all strings to JSON using JSON.parse(which blocks the event-loop).

var arr = ["{...}", "{...}", ... ]      //input array

Here is the first approach(may keep the event loop blocked for some time):

var newArr = arr.map(function(val){
  try{
    var obj = JSON.parse(val);
    return obj;
  }
  catch(err){return {};}

});

The second approach was using async.map method(Will this be more efficient compared to the first approach?):

var newArr = [];
async.map(arr, 
  function(val, done){
    try{
        var obj = JSON.parse(val);
        done(null, obj);
      }
      catch(err){done(null, {});}
  }, 
  function(err, results){
    if(!err) 
      newArr = results;
  }
);

If the second approach is same or almost same then what is efficient way of doing this in node.js.

I came across child processes, will this be a good approach for this problem?

Aman Gupta
  • 3,627
  • 4
  • 40
  • 64

2 Answers2

1

I don't think async.map guarantees a non-blocking handling of a sync function. Though it wraps your function with an asyncify function, I can't find anything in that code that actually makes it non-blocking. It's one of the problems I've encountered with async in the past (but maybe it's improved now)

You could definitely handroll your own solution with child processes, but it might be easier to use something like https://github.com/audreyt/node-webworker-threads

Creynders
  • 4,573
  • 20
  • 21
  • 1
    The documentation of [`async`](http://caolan.github.io/async/index.html) states: _"Async does not guard against synchronous iteratees for performance reasons"_. So it doesn't "asynchronize" the iteratee function by itself. – robertklep May 08 '17 at 12:26
  • Thanks @robertklep so apparently the problem still remains. Very weird, IMO. – Creynders May 08 '17 at 12:27
  • Well, the whole point of `async` is to handle _asynchronous_ workflows ;D If it's being used in a synchronous fashion, you might as well not use it at all (although the documentation does show some cases in which sync and async code is mixed). – robertklep May 08 '17 at 12:29
  • Yeah sure, but as with Promises there are tons of situations in which you don't know whether you're working with blocking or non-blocking code. Fortunately promises _do_ ensure it will finish async. Zalgo is a mean mofo to get bitten by. – Creynders May 08 '17 at 12:33
  • Particularly when using `async`, I think it's generally pretty clear if code is async or not. But yes, promises take away any confusion on that part :) – robertklep May 08 '17 at 12:38
  • Certainly. If you're the writer of the functions you're consuming. For instance: I've written a hooking lib, in which case I have no idea what kinds of functions I'm executing. All I know is that in the end everything needs to finish asynchronously or all hell breaks loose. https://github.com/keystonejs/grappling-hook – Creynders May 08 '17 at 12:44
  • What I mean that as a user of `async` one should know whether or not the code they are calling is async or not: it takes a callback, or it returns a promise, for instance. As an implementor of a library that may accept both sync or async functions, it's a different story (esp when functions return promises, where you can't rely on whether or not the function takes a callback argument). – robertklep May 08 '17 at 12:56
  • Thanks. Will look into webworker and see how things go. – Aman Gupta May 08 '17 at 15:33
1

use async.map but wrap the callback in setImmediate(done)

I find the async functions quite convenient but not very efficient; if the mapped computation is very fast, calling done via setImmediate only once every 10 times and calling it directly otherwise will run visibly faster. (The setImmediate breaks up the call stack and yields to the event loop, but the setImmediate overhead is non-negligible)

Andras
  • 2,995
  • 11
  • 17