0

I am trying to open a series of url in an array in a headless browser using puppeteer. It works fine when one url is used, but when a loop is used, puppeteer throws an exception.

$("a").each(async (index,element)=>{
    if($(element).attr("target") == "_blank"){
        //urls are valid
        let url = "https:"+$(element).attr("href");
        let page = await browserPage.goto(url).then((response)=>{
            console.log(response.status())
        },(error)=>{
                console.log(error)
        })

    }
})

An ERR_ABORTED exception is thrown for each element of the array

Is this some kind of rate limiting problem? if so how can i work around it?

Noble Eugene
  • 523
  • 1
  • 5
  • 15
  • 1
    First of all, [probably don't use cheerio with puppeteer unless you have a really good reason for it](https://serpapi.com/blog/puppeteer-antipatterns/#using-a-separate-html-parser-with-puppeteer). It'll just lead to confusion and bugs like this. Second of all, the pattern you're using of async in a foreach means that all of the `goto`s will run simultaneously on the same page, resulting in disaster. – ggorlen Jan 07 '23 at 19:34
  • Hi @ggorlen, the cheerio is not being used with puppeteer, i was acutually using it with axios to crawl static pages. While puppeteer was intended to be used differently. I will try to run the loop non asynchronously – Noble Eugene Jan 07 '23 at 20:14
  • 2
    If you read the dupes, the solution isn't to run it non-asynchronously. In fact, that's impossible because Puppeteer has an async API. The solution is to use a non-`.each`/`.forEach` loop. OK, if Cheerio is used with axios to get URLs, no problem. – ggorlen Jan 07 '23 at 20:18
  • 2
    Hi @ggorlen, i rewrote the code to use a for loop instead of an each loop and it works great now. Thanks a bunch – Noble Eugene Jan 07 '23 at 21:23

0 Answers0