0

I'm running into a timeout error when using Puppeteer to navigate to a heavy website with lots of videos. I've increased the timeout to 690000 ms using page.goto(), and set the default navigation timeout to 0 using page.setDefaultNavigationTimeout(), but I'm still getting the following error:

CAUGHT AN ERROR TimeoutError: Navigation timeout of 90000 ms exceeded at LifecycleWatcher._LifecycleWatcher_createTimeoutPromise (C:\Users\Aliyan Malik\Desktop\Moniter System CRC\node_modules\puppeteer-core\lib\cjs\puppeteer\common\LifecycleWatcher.js:162:12)

Here's my code:

const express = require("express");
const app = express();
const puppeteer = require("puppeteer");

let timer;

async function checkSite() {
  let browser;
  try {
    console.log("TEST SCRIPT STARTED...");
    browser = await puppeteer.launch({
      headless: false,
      // timeout: 990000,
      // protocolTimeout: 990000,
    });
    const page = await browser.newPage();
    await page.setDefaultNavigationTimeout(0);
    await page.goto("https://www.crcmedia.com/", {
      timeout: 690000,
      waitUntil: "domcontentloaded",
    });

    // await page.waitForNavigation({
    //   timeout: 690000,
    //   waitUntil: "domcontentloaded",
    // });
    await page.waitForSelector("custom-gallery");
    const videoTags = await page.evaluate(() => {
      const customGallery = document.getElementsByTagName("custom-gallery")[0];
      const shadowHost = customGallery.shadowRoot;
      const videoElements = shadowHost.querySelectorAll("video");
      return Array.from(videoElements);
    });
    console.log("VIDEO TAGS", videoTags);

    for (let i = 0; i < videoTags.length; i++) {
      console.log("I", i);
      const video = videoTags[i];
      await video.hover();
      const isHovered = await page.evaluate(() => {
        const video = document.querySelector("video");
        return video === document.activeElement;
      });

      await video.click();
      const isClicked = await page.evaluate(() => {
        const video = document.querySelector("video");
        return video.paused === false;
      });

      if (isHovered && isClicked) {
        console.log(`Video ${i + 1} interactions working properly`);
      } else {
        console.log(`Video ${i + 1} interactions not working properly`);
      }
    }
  } catch (error) {
    console.error("CAUGHT AN ERROR", error);
  }

  if (browser) {
    await browser.close();
  }
}

app.get("/", (req, res) => {
  res.send("Site monitoring is running");
  //   timer = setInterval(checkSite, 60 * 60 * 1000); // Run checkSite function every hour
  checkSite().catch((error) => console.error("ERROR RUNNING CHECKSITE", error));
});

app.listen(3000, () => {
  console.log("Server is running on port 3000");
  checkSite().catch((error) => console.error("ERROR RUNNING CHECKSITE", error));
});

Is there something else I can do to increase the timeout or make the navigation more efficient? Any help would be greatly appreciated. Thank you!

"I increased the timeout setting for my Puppeteer script to 690000 milliseconds to account for the website's heavy video content. I also set the setDefaultNavigationTimeout method to 0 to disable the default timeout. Despite these adjustments, I continue to receive the Navigation timeout of 90000 ms exceeded error."

Jacob Cooper
  • 1
  • Please don't set timeouts to 0. That stifles useful errors. If a nav takes more than 2-3 minutes, it's probably never going to load and you'll need the error to debug it. Which timeout is failing? A [mcve] would be nice--all of the express stuff seems irrelevant. What are you trying to achieve here? There may be an easier way to get that result without all the resources the site loads. – ggorlen Apr 19 '23 at 19:28
  • The code line "await page.waitForNavigation({ timeout: 690000, waitUntil: "domcontentloaded" });" is causing a delay and resulting in an error. I am configuring an express server to deploy my website on Heroku and continuously run tests. My website, www.crcmedia.com, has many videos in a custom-designed Wix feature, and I want to test hover and click events to ensure they work properly. As a beginner with Puppeteer, I would appreciate your help in optimizing my code and reviewing my site. Thank you for your assistance. – Jacob Cooper Apr 19 '23 at 20:21
  • Oh, well that's commented out and shouldn't be there. `goto` already waits for navigation. If you wait for it a second time, that'll time out since the page is already loaded. So the solution is to keep the code as is, and remove the commented out `waitForNavigation` since it's misleading and you don't want to be tempted to use it. – ggorlen Apr 19 '23 at 20:24
  • Thank you but I am getting this error now "CAUGHT AN ERROR Error: Evaluation failed: TypeError: Cannot read properties of null (reading 'querySelectorAll')" As my videos containers are in the shadow root, I am able to get them when I do this with chrome console. Any idea about this? how can we target that? – Jacob Cooper Apr 19 '23 at 20:53
  • Even if that worked, [you can't return DOM nodes from evaluate](https://stackoverflow.com/questions/46377955/puppeteer-page-evaluate-queryselectorall-return-empty-objects). Try `page.$$("pierce/custom-gallery video")` to get your video ElementHandles. I also suggest updating your post to match this since navigation is no longer the issue. – ggorlen Apr 19 '23 at 20:59
  • See [this post](https://blog.appsignal.com/2023/02/08/puppeteer-in-nodejs-common-mistakes-to-avoid.html#making-unnecessary-calls-to-waitfornetworkidle-or-waitfornavigation) regarding your first misconception and [`pierce/`](https://pptr.dev/guides/query-selectors#pierce-selectors-pierce) in the pupp docs. – ggorlen Apr 19 '23 at 21:07
  • Can you please join a call to look into the issue? – Jacob Cooper Apr 19 '23 at 21:23
  • If you want private assistance, you're welcome to [hire me](https://www.codementor.io/@ggorlen) for consulting work. Otherwise, the idea with Stack Overflow is to curate a resource that helps the community, so everything would be public. – ggorlen Apr 19 '23 at 21:27

0 Answers0