How do I send bulk HTTP GET requests using Axios, for example:
let maxI = 3000;
let i = 0;
do{
i = i + 1 ;
await exampleUrl = axios.get(`https://hellowWorld.com/${i}`);
} while (i < maxI);How will I be able to receive the data from all the provided URLs and can this be merged into a single variable? And how can I make sure that this gets executed quickly?
I know about axios.all, but I don't know how to apply it in my case.
Thank you in advance.
You can do something like this but be careful, servers will reject your request if you make them in bulk to prevent DDOS and this also doesn't guarantee that all the requests would return successfully and you will receive all the data, here is the snippet for it:
import axios from "axios";
const URL = "https://randomuser.me/api/?results=";
async function getData() {
const requests = [];
for (let i = 1; i < 6; i++) {
requests.push(axios.get(URL + i));
}
const responses = await Promise.allSettled(requests);
console.log(responses);
const result = [];
responses.forEach((item) => {
if (item.status === "rejected") return;
result.push(item.value.data.results);
});
console.log(result.flat());
}
getData();
AFAIK, it is impossible to increase the speed/reduce the time taken to complete your batch requests unless you implement a batch request handling API on server which would reduce both number of requests handled by the server and the number of requests made by the browser. The solution I gave you is just to demonstrate how it could be done from client side but your approach is not an optimal way to do it.
There is a different limit for number of parallel requests that can be made to a single domain for different browsers because of which we cannot reduce the time taken to execute queries.
Please read through these resources for further information they will be of great help:
