How to use vanilla Node.js to run 500 requests, 20 in parallel

Question

I have this vanilla Node.js code:

const http = require('http');

const host = 'example.com';
const path = '/';

let i = 0;

const run = () => {

  console.log(i++);

  return new Promise(resolve => {

    const req = http.request({host, path}, res => {
      res.pipe(process.stdout);
      res.once('end', resolve);
    });

    req.end();

  });

};

async function doall() {
  for (let i = 0; i < 50; i++) {
    await Promise.all(new Array(10).fill(null).map(run));
  }
}

const now = Date.now();

console.log('Starting up.');

doall().then(_ => {
  console.log('done after:', Date.now() - now, 'millis');
});

// the end

this works - it runs 50 sets of 10... except the problem is that, all 10 complete, then the next 10 start, then the next 10 complete. So there are moments when 0 requests are in progress, between each set.

Is there some way using vanilla Node.js and promises to replicate async.eachLimit(new Array(500), 20, ...)?

so you want to queue the next one after each fail/success of a single request? — Icepickle, Sep 20 '19 at 22:23
yeah that's right, if there are fewer than 20 requests currently in progress — , Sep 20 '19 at 22:24
Solutions to this problem [`mapConcurrent()`](https://stackoverflow.com/questions/41028790/javascript-how-to-control-how-many-promises-access-network-in-parallel/41028877#41028877) and [`pMap()`](https://stackoverflow.com/questions/33378923/make-several-requests-to-an-api-that-can-only-handle-20-request-a-minute/33379149#33379149) and [`runN()`](https://stackoverflow.com/questions/48842555/loop-through-an-api-get-request-with-variable-url/48844820#48844820). — jfriend00, Sep 20 '19 at 22:50
@jfriend00 I tried to make it clear that this question was how to do this with vanilla JS, so we don't have to import dependencies etc — , Sep 23 '19 at 18:06
The question yours is marked a duplicate of contains code (the `mapConcurrent()` function) that you could just copy and use. I've also provided you links to three other implementations that you can likewise copy and use. There are plenty of sources already existing in other answers. I've given you FOUR options that don't rely on an outside library. — jfriend00, Sep 23 '19 at 18:24

Nicholas Tower · Answer 1 · 2019-09-20T22:48:22.470

1

Here's the solution i came up with.:

function run() {
  // mocking this out with a random timeout for the purposes of the snippet.
  return new Promise((resolve) => setTimeout(resolve, Math.random() * 300))
}

function doAll(totalCount, maxConcurrent) {
  return new Promise((resolve) => {
    let started = 0;
    let settled = 0;

    // Every time we start a task we'll do this
    const enqueue = () => {
      started++;
      console.log('starting', started)

      // I assume you want errors to count as done and move on to the next
      run().then(onSettled, onSettled);   
    }

    // Every time we finish a task, we'll do this:
    const onSettled = () => {
      settled++;
      console.log('finished', settled)
      if (started < totalCount) {
        enqueue();
      }
      if (settled === totalCount) {
        resolve();
      }
    };

    // Start the initial batch going.
    while (started < maxConcurrent && started < totalCount) {
      enqueue();
    }
  })
}

doAll(10, 5)
  .then(() => console.log('all done'));

edited Sep 20 '19 at 22:48

answered Sep 20 '19 at 22:40

Nicholas Tower

72,740
7
86
98

It doesn't look like this has any means of communicating back errors or accumulating results in order. – jfriend00 Sep 23 '19 at 18:28
`It doesn't look like this has any means of communicating back errors or accumulating results in order.` Correct, it does not. If those are needed then this could would not be enough. `I also don't see how it ever decrements started` It doesn't; that number always increases. Starting the next one is done when a promise settles via the code `if (started < totalCount) { enqueue(); }` – Nicholas Tower Sep 23 '19 at 18:32
Yeah, the second part was removed from my comment because I then discovered how you were doing that. – jfriend00 Sep 23 '19 at 18:33
Does mine accumulate results and store order? i believe so. Is mine the prettiest in the land? You be the judge. – Sep 25 '19 at 07:02

score 0 · Answer 2 · 2019-09-25T07:00:35.840

0

This seems to work:

const http = require('http');

const host = 'example.com';
const path = '/';

let i = 0;

const makeRequest = () => {

  console.log(i++);

  return new Promise(resolve => {

    const req = http.request({host, path}, res => {
      res.pipe(process.stdout);
      res.once('end', resolve);
    });

    req.end();

  });

};

const run = (v) => {

  v.count++;
  v.total++;

  if (v.total > v.max) {
    return;
  }

  return makeRequest().then(_ => {

    v.count--;

    if (v.count < v.concurrency) {
      return run(v);
    }

  });
};

async function doall() {

  const v = {
    count: 0,
    total: 0,
    max: 500,
    concurrency: 20
  };

  const initialSet = new Array(v.concurrency).fill(null);

  return Promise.all(initialSet.map(_ => run(v)));

}

const now = Date.now();

console.log('Starting up.');

doall().then(_ => {
  console.log('done after:', Date.now() - now, 'millis');
});

Now 20 run in parallel in any moment, and 500 run in total. So now it behaves more like async.eachLimit(new Array(500).., 20, ...).

edited Sep 25 '19 at 07:00

answered Sep 20 '19 at 22:33

but you are rerunning the same `v`, now don't you? – Icepickle Sep 20 '19 at 22:36
I am not returning `v` anywhere, what do you mean? – Sep 20 '19 at 22:38
No, not returning, however you `run(v)` where `v` is an argument of the parent function; say you use this [api](https://dog.ceo/dog-api/documentation/) to get the breedlist, can you a picture for all of the breeds with your code and arguments? – Icepickle Sep 20 '19 at 22:41
@Icepickle `v` has nothing to do with the request. It merely tracks how many connections are running and how many are still waiting to run. What you seem to be complaining about is the `path` variable which is a constant. But that can easily be fixed by making it an array and popping from it on each new request – slebetman Sep 21 '19 at 00:30
@slebetman I wasn't complaining, just wondering. I am not fond of modifying input arguments, and reusing them could be a bit problematic :) – Icepickle Sep 22 '19 at 15:12
1

@Icepickle But he's creating a new object on each function call so there is zero problems with reuse. The code is completely re-entrant. It would be an issue if `v` is global but since it's passed in as input each call gets its own state variable. – slebetman Sep 22 '19 at 16:22
MrCholo approves these messages – Sep 25 '19 at 07:04

How to use vanilla Node.js to run 500 requests, 20 in parallel

2 Answers2