0

I have the following code:

const puppeteer = require('puppeteer');
const cheerio = require('cheerio');

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(<my_url>, { waitUntil: 'networkidle0' });

// Get the content of the first page
const page1 = await page.content();
let $ = cheerio.load(page1);
let items = $($('div[id*=document]')).toArray();

// Go to another page
const buttonToPage2 = await page.$('li[data-page="2"]');
await buttonToPage2.click();

// Get the content from the second page
const page2 = await page.content();
let $ = cheerio.load(page2);
items = items.concat($($('div[id*=document]')).toArray());

Doesn't it make sense? It seems I'm not getting the right content from the page 2. Anyway, do you know a better way to reach that?

smartmouse
  • 13,912
  • 34
  • 100
  • 166
  • have you checked into `page.waitForNavigation` like described here? https://stackoverflow.com/questions/46948489/puppeteer-wait-page-load-after-form-submit – madeyejm Sep 23 '19 at 13:32
  • I have tried to use both `await page.click('li[data-page="2"]', { waitUntil: 'networkidle0' }); const page2 = await page.content();` and `await page.click('li[data-page="2"]', { waitUntil: 'networkidle0' }); const page2 = await page.content();` but nothing has changed... – smartmouse Sep 23 '19 at 13:46
  • In both cases I get content not completely loaded... – smartmouse Sep 23 '19 at 13:55
  • 1
    It is a problem related to the fact that I'm facing with a single-page-application: https://github.com/GoogleChrome/puppeteer/issues/1412#issuecomment-379428743. I solved using `.waitForSelector`. – smartmouse Sep 23 '19 at 14:34
  • Too much left to the imagination here. It's almost impossible to help without seeing the actual pages. Voting to close as lacking a [mcve]. Also, using Cheerio with Puppeteer doesn't make much sense to me--Puppeteer already has a suite of selectors and had parsed the live HTML--no need to re-parse it with a separate tool. – ggorlen Nov 26 '22 at 01:48

0 Answers0