0

I am fetching over 7 million records from an API using fetch-node in a node js script.

The records each have a sequential id starting from 1, so I using a for loop to access each record. The main issue I am having is speed!

Initially I did one a time, but I have tried using promise all to do them in batches (see code below) - which has greatly improved the speed, but I think there must be a more efficient way to do this.

(async () => {

    const start = 1;
    const records_to_fetch = 7000000;
    const finish = start + records_to_fetch + 1;

    for(i = start; i < finish; i += 5) {
        
        await Promise.all([
            getHistory(i),
            getHistory(i+1),
            getHistory(i+2),
            getHistory(i+3),
            getHistory(i+4),
        ]);
        
    };
}

})();

Without the promise.all, the API can't handle the volume of requests when doing it in batches, however as each function call is independent from each other I don't need all 5 to finish before starting the next one. Ideally I would have 5 "slots" (or more in reality) so I can limit the number of concurrent calls, but when one finishes the next id is called.

Part of the problem is that I don't even know how to phrase my question when searching for an answer! I am hoping someone can understand what I'm trying to achieve and give me some help!

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
joecampbelluk
  • 71
  • 1
  • 2
  • 5
  • https://futurestud.io/tutorials/node-js-run-async-functions-in-batches – Manos Kounelakis Dec 15 '20 at 09:42
  • Also are you sure that the API you retrieving the data from doesn't have an endpoint that does that ? Seems overkill both for the consumer (you) and the producer (API). – Manos Kounelakis Dec 15 '20 at 09:44
  • Hi Manos, thanks for your reponse! Sadly the only endpoint that has the data I need can be accessed via the individual's id only. Forgive my lack of knowledge (I'm pretty new to coding), but how would I run a for loop using a promise pool? It seems to require an array, but mine would be over 7million items which seems like overkill.... – joecampbelluk Dec 15 '20 at 09:54

0 Answers0