1

We have been using page.goto() with waitUntil & timeout options of Puppeteer to download the content of the page. But recently we observed that for some urls returning dynamic content(developed using Angular, ReactJs, VueJs, etc.) it is partially returning the page content. Seems like waitUntil doesn't have mechanism to sense that entire page content is loaded. We either need to add await sleep() or set a higher default timeout in networkdidle0 option parameter to page.goto(). Since the page content is dynamically generated, we cannot preset the timeout or sleep value. We are also using page.content() which can be used to download the page content. Is there any option parameter available which can help us to get the full content of the page? Or is there any event linked to the page.content() which gets triggered once it fetches entire page content?

pravin.salgaonkar
  • 11
  • 2
  • The concept of "complete" is, as you say, somewhat nebulous, since JS can run at arbitrary times and fetch more resources and manipulate elements. You get to decide what constitutes "complete" and pull the content at that time. Puppeteer provides various tools to do this: `goto`/`waitForNavigation` can release at `load`, `domcontentloaded`, `networkidle2`, `networkidle0`. Beyond that, there's `waitForSelector`, `waitForFunction`, `waitForRequest`, and if all else fails, `waitForTimeout`, although you almost never need to use that if you're careful and precise. – ggorlen Jun 07 '22 at 16:23
  • If you want a one-size-fits-all solution, there isn't any because every site is unique. If you pick any value of `waitForTimeout`, there's always a site that might take longer to load than that value. And if your value is too high, then you're going to slow all the fast pages unnecessarily. `networkidle2` plus a short timeout is an OK approximation, but any approach will have inaccuracies. It's best to describe your use case a bit clearer so you can manage the tradeoffs appropriately. But yeah, Puppeteer can't possibly have a `waitUntilLoaded` that magically works on all websites ever. – ggorlen Jun 07 '22 at 16:28

0 Answers0