How do you map generator results concurrently?

Question

How can you map the results of async calls in a generator, concurrently?

var generator = (function *() {
  var lotsOfThings = yield asyncTask.call(generator);

  var mappedResults = yield lotsOfThings.map(thing => {
    // fails with a syntax error unless you do a `for…of` loop, but also doesn’t run concurrently regardless
    return (yield asyncTask.call(generator, thing));
  });

  // then do something with mappedResults
})();

generator.next();

function asyncTask(…query) {
  somethingAsync(…query, (err, res) => this.next(res));
}

Also, even in a regular for...of loop, you can’t run each asyncTask concurrently. yield will cause a pause between each task, essentially making a synchronous AJAX request. Ideally you’d want it to work like it does with promises, like this paradigm:

// these tasks will run concurrently (unlike the above example)
let promises = someThings.map(thing => {
  return new Promise((resolve, reject) => {
    somethingAsync((err, res) => {
      resolve(res);
    });
  });
});

Promise.all(promises).then(/* do stuff */);

The promise approach can get hairy 'cause of all the nesting, but the benefit is that the async tasks can run concurrently… whereas the generators look nice, but looping through tasks is not concurrent. Any ideas?

Generators are iterative on their nature, why would you expect to run them in parallel? From client's perspective, genererators are just something which you call next() on, or use in a loop. — Mikhail Shilkov, Sep 15 '15 at 11:57
Yes, you really should just use promises for your asynchronous tasks, not callbacks - you've got exactly this `Promise.all` for free. Then use a "coroutine" library (also part of many promise libs) that allows you to run generators with them. **Don't write this code yourself**. — Bergi, Sep 15 '15 at 14:37
@Bergi, thanks. This may be out of the scope of the question, but do you know of a coroutine library off the top of your head that can give the syntactic sugar of the generators and the concurrency of promises? I may be asking for a unicorn... — Josh Beam, Sep 15 '15 at 17:54
No, the coroutine library does only give you the syntactic sugar, the concurrency is achieved through `Promise.all`. Of course, some libs like Q, when, Bluebird etc give you both functions. Also most pure-coroutine libs like `co` are able to await `yield`ed collections (like arrays), neatly abstracting the callback counter. — Bergi, Sep 15 '15 at 18:45
@Bergi, thanks, it looks like the functionality I was looking for is to be able to yield a collection of items in parallel, which looks like it possible through `co` (if each item is a promise). — Josh Beam, Sep 15 '15 at 18:48

score 1 · Answer 1 · answered Sep 16 '15 at 10:11

1

I tried to sketch something similar without third-party libraries:

// Async runner
function async(generator){
  var process = function(result){       
    if (result.done) {
      return;
    }
    (Array.isArray(result.value) ? Promise.all(result.value) : result.value).then(function(value){
      process(sequence.next(value));
    });
  };

  var sequence = generator();
  var next = sequence.next();
  process(next);
};

// Generator function
var generator = function* () {
  var list = yield getList();
  console.log(list); // outputs [1, 2, 3, 4, 5]

  var details = yield list.map(p => getDetails(p));    
  console.log(details); // outputs [11, 12, 13, 14, 15]
}

// Your async requests go here
function fakeAsync(f) {
  return new Promise(function(resolve, reject) {
    setTimeout(function() {
      f(resolve);
    }, 500);
  });
}

function getList() {
  return fakeAsync(function(resolve) {
    resolve([1, 2, 3, 4, 5]);                  
  });
}

function getDetails(i) {
  return fakeAsync(function(resolve) {
    resolve(i + 10);                  
  });
}

async(generator);

Is it what you were trying to achieve?

answered Sep 16 '15 at 10:11

Mikhail Shilkov

34,128
3
68
107

This solution lacks error handling. Which is one reason why you should just use a library (which is necessary to polyfill `Promise.all` anyway) – Bergi Sep 16 '15 at 11:58
@Mikhail, it may be what I was trying to achieve. It looks like the underlying paradigm is that to have these things run in parallel, you basically need to yield to a promise or a set of promises. Am I on the right track by saying that? – Josh Beam Sep 16 '15 at 18:00
1

@JoshBeam Yes, you are right. Basically, whatever you yield has to be understood by the code which runs through generator. If you need parallel promises to run, you yield them and resolve all. – Mikhail Shilkov Sep 17 '15 at 10:00
@Bergi True. This is not a production-ready code, just illustration of the concept. It's ok to use libraries but it's even better to understand how they do what they do. – Mikhail Shilkov Sep 17 '15 at 10:03

Josh Beam · Accepted Answer · 2015-09-16T18:03:29.560

0

The answer I was looking for can be implemented with co.

co(function* () {
  // maybe we need a token from our API before we can do anything
  let token = yield new Promise((resolve, reject) {
    getToken(token => resolve(token));
  });

  // these run in parallel
  var queries = yield [
    request('/route', token),
    request('/another-route', token)
  ];

  // [[], []] becomes [...]
  return _.flatten(queries);
});

// our async request function returns a promise
function request(route, token) {
  return new Promise((resolve, reject) => {
    somethingAsync(route, token, (res) => {
      resolve(res);
    });
  });
}

edit: changed somethingAsync to not be thenable, as in my actual case, it's a call to a 3rd party API that does not already return a promise.

edited Sep 16 '15 at 18:03

answered Sep 16 '15 at 05:53

Josh Beam

19,292
3
45
68

Avoid the [`Promise` constructor antipattern](http://stackoverflow.com/q/23803743/1048572) in your `request` method, though – Bergi Sep 16 '15 at 08:57
@Bergi yeah thanks. In his case, somethingAsync is referring to, say, a third party API call that doesn't already return a promise, so I think I'm safe wrapping it in one since it needs to return a promise. Is that right? – Josh Beam Sep 16 '15 at 17:57
Yeah, in that case it would, but the code in your answer does not look like that. `somethingAsync` seems to at least return a thenable (i.e. a "promise" whose implementation you don't trust), so you should be doing `return Promise.resolve(somethingAsync(route, token));` – Bergi Sep 16 '15 at 18:00
@Bergi ah shoot you're right, yes that was a typing mistake on my part. In my actually context, somethingAsync would actually take a callback, and not be thenable. – Josh Beam Sep 16 '15 at 18:02

How do you map generator results concurrently?

2 Answers2