Parallel asynchronous iteraor - is it possible?

Question

Right now I have the following code:

import axios from 'axios'

const urls = ['https://google.com', 'https://yahoo.com']

async function* requests() {
  for (const url of urls) {
    yield axios.get(url)
  }
}

;(async () => {
  for await (const n of requests()) {
    console.log(n.config.url) // prints https://google.com and then https://yahoo.com
  }
})()

As is, the requests won't block the single thread of node, but they will happen in sequence. I'm wondering if it would be possible to change the code to force parallelism.

would [Promise.all](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/all) solve your problem? — stranded, Feb 06 '20 at 14:31
Not sure what you mean. How would I use Promise.all to parallelize things? I also need to use the response as they are available because we have a memory constrain. — , Feb 06 '20 at 14:35
in your case, if you `await Promise.all(requests())` it will return you an array with the resolved promises of `axios.get()` So it will run all requests in parallel, and Promise.all() resolves when every Promise inside the iterator has resolved — stranded, Feb 06 '20 at 14:39
That's not good enough, I can't hold every one of those responses in memory as I stated before. I need to use them as they are available, like a stream. — , Feb 06 '20 at 14:49
You can batch them, instead of every request, iterate every n requests. Or use rxjs — SrThompson, Feb 06 '20 at 14:54
Does this help @gos3 : https://stackoverflow.com/questions/42768461/how-to-run-generator-functions-in-parallel — scgough, Feb 06 '20 at 15:08
Also - this script `co` seems to be a nice approach https://github.com/tj/co#arrays — scgough, Feb 06 '20 at 15:11

score 3 · Accepted Answer · answered Feb 06 '20 at 16:20

The "simpler" no-deps way would be to batch them and yield every batch with Promise.all

import axios from 'axios'

const urls = [
  'https://jsonplaceholder.typicode.com/todos/1', 
  'https://jsonplaceholder.typicode.com/posts/1',
  'https://jsonplaceholder.typicode.com/users/1',
  'https://jsonplaceholder.typicode.com/comments/1'
]

async function* requests(batchSize = 1) {
  let batchedRequests = [];
  for (const url of urls) {
    batchedRequests.push(axios.get(url));
    if (batchedRequests.length === batchSize) {
      yield Promise.all(batchedRequests);
      batchedRequests = [];
    }
  }
  if (batchedRequests.length) { //if there are requests left in batch
    yield Promise.all(batchedRequests);
  }
}

;(async () => {
  for await (const batch of requests(2)) {
    batch.forEach(n => console.log(n.config.url)) // prints https://google.com and then https://yahoo.com
  }
})()

You can use rxjs to achieve similar results, with the advantages that observables have in terms of flexibility, but it's another library and can be more complex if you're not familiar with reactive streams. Here is a detailed post I found on the topic: https://medium.com/@ravishivt/batch-processing-with-rxjs-6408b0761f39

Parallel asynchronous iteraor - is it possible?

1 Answers1