How to run multiple(>200) http requests in node.js without getting ENOTFOUND error

Question

I am trying to scrape a big list of pages from a website. Here is the axios request which I make :

 Promise.all(urls.map((url, index) => axios(url))).then(function (values) {
        ........
 });

The urls object just have a bunch of urls > 13000. If i keep that number to 200 the whole code works fine. But when I put in the entire object i get this:

errno: 'ENOTFOUND', code: 'ENOTFOUND', syscall: 'getaddrinfo',

I have read all the existing answers but nothing helped so please!

I would suggest not to hit so many requests at a time , try hitting them in some sort of pagination. May be your server resources are consumed upon hitting such large requests. Also getaddrinfo is a DNS issue. — Prabhjot Singh Kainth, Dec 27 '19 at 05:39
yes I've tried creating a new promise wrapper for each request with a good amount of delay but still same error. The website I'm scraping is a pretty big firm so I'm quite sure its a node issue. — Rahul Purohit, Dec 27 '19 at 05:44

Himanshu Joshi · Accepted Answer · 2022-06-28T09:53:13.927

This you need to hit the server in chunks of request like 100 at a time. Send as every time you try to call API it goes to the DNS server and due to so. many requests it either sometimes discard or not able to get your host address in the registry. So try using dnscache from npm. https://www.npmjs.com/package/dnscache.

Also for more context check https://github.com/request/request/issues/2536

Do require('dnscache')({ enable: true }) in a proxy file or where your code lies of calling API. This will make DNS package the first check-in local cache for the host and then go to DNS if not found locally.

How to run multiple(>200) http requests in node.js without getting ENOTFOUND error

1 Answers1