2

I'm trying to crawl and scrape some sites to extract some links. I expect to see all the hrefs printed to my console. But, instead, I get the following error.

out [ Promise { } ] (node:15908) UnhandledPromiseRejectionWarning: Error: Protocol error (Page.navigate): Target closed.

What am I doing wrong?

This answer says the error message means browser.close() has already executed by the time I call my pageFunction.

But I'm using async await and apparently the browser is still closing on me.

How can I fix this error?

const domains = [...]

const pageFunction = async $posts => {
  const data = [];
  await $posts.forEach( $post => {
    data.push( $post.href );
  });
  return data;
}

(async () => {
  // start browser
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // loop over domains
  const out = await domains.slice(-1).map( async domain => {
    const url = [ 'http', domain, ].join(joiner2);
    await page.goto( url, waitUntilLoad, );
    const hrefs = await page.$$eval( 'a', pageFunction, );
    return hrefs;
  });

  // log hrefs
  console.log( 'out', out, );
  await browser.close();
})();

Also, note: When I use:

  const pageFunction = async $posts =>
    await $posts.map( $post => $post.href )

The error goes away. But I also get no output either. I would expect the console to log the hrefs but instead it logs nothing.

FWIW: Here is the question I wrote yesterday on the same piece of code. I modified the code by only doing one URL for now slice(-1) instead of the whole list. And now I get the above error instead of the one I describe in yesterday's question.

Let Me Tink About It
  • 15,156
  • 21
  • 98
  • 207
  • My two cents here. That error could also mean that the page (or chromium) crashed. – hardkoded Dec 06 '19 at 13:25
  • I'm wondering whether `await domains.slice(-1).map( async domain => {` is doing what you expect. Are you sure you can make a mapping async? – hardkoded Dec 06 '19 at 13:30

1 Answers1

5

I'm quite sure it is because inside the anonymous function you are awaiting for the result of map, which instantly returns an array of Promises. This means your code execution proceeds to closing the browser.

You should try the following:

const promises = domains.slice(-1).map( async domain => {...});
const out = await Promise.all(promises);
Alessandro Vago
  • 66
  • 1