I am making http requests via node.js and axios freezes after some hundred iterations.
I already used some console.logs to narrow down the problem but I have no idea how to find out what's the underlying problem and fix it... any hints on how to debug this would be very much appreciated.
Context: in this worker thread a getTokenData function is called, that's where the axios http request is.
//worker.js
parentPort.on("message", async (data: dataFromParentPort) => {
// ... variable definitions
for (let token_index = 0;
token_index < 3000;
token_index++) {
console.log(`Looping through HTTP requests... request ${token_index}/${supply}`);
let url: string = baseURI + token_index.toString();
await getTokenData(url, token_index)
.then((json: any) => {
console.log("Promise resolved, got status " + json.status)
// do stuff with json
})
.catch((rej) => {
console.log("[Error] Catched this: " + JSON.stringify(rej))
})
console.log(`${guildId}/${address}: Token #${token_index} cached.`)
}
function getTokenData(_url: string, _tokenId: number) {
return new Promise( (resolve, reject) => {
console.log("Init Http Request...")
if ((_tokenId % 50 == 0)) {
console.log("Pausing...");
setTimeout(() => {
console.log("Axios is starting request.")
axios.get(_url)
.then(response => {
console.log("Axios got a response: " + response.data.name)
resolve({
status: 1,
data: response.data
});
})
.catch((error: any) => {
console.log();
console.error("Error: " + error.message);
reject({
status: 0,
data: error
});
})
}, 3000)
}
else {
console.log("Axios is starting request.")
axios.get(_url)
.then(response => {
console.log("Axios got a response: " + response.data.name)
resolve({
status: 1,
data: response.data
});
})
.catch((error: any) => {
console.log();
console.error("Error: " + error.message);
reject({
status: 0,
data: error
});
})
}
})
}
In my desperation I even wrapped the call in a setTimeout command because I thought the reason for axios "freezing" might be too many requests to the ipfs gateway per certain intervals or something like that. It seems not to be the case, because sometimes axios freezes after a few hundred, sometimes after roughly 1k-1,5k requests.
This is how the console.logs just stop without any error messages.
Looping through HTTP requests... request 1469/3333
Init Http Request...
Axios is starting request.
Axios got a response: Angry Ape #1469
Promise resolved, got status 1
123/0x77640cf3f89a4f1b5ca3a1e5c87f3f5b12ebf87e: Token #1469 cached. Looping.
Looping through HTTP requests... request 1470/3333
Init Http Request...
Axios is starting request.
Axios got a response: Angry Ape #1470
Promise resolved, got status 1
123/0x77640cf3f89a4f1b5ca3a1e5c87f3f5b12ebf87e: Token #1470 cached. Looping.
Looping through HTTP requests... request 1471/3333
Init Http Request...
Axios is starting request.
I wonder how I could analyze this further and find the actual problem... also wonder wether this is an axios/node.js thing and maybe another request method could do better?
