1

I am able to navigate to a page using puppeteer but afterwards the page.evaluate is not returning any response. Further, I am unable to debug inside the page.evaluate either. I run the script in debug mode (node debug filename.js), use sb(15) to set the breakpoint on line 15, press c to continue, wait for the page to load, then enter 'repl'. Now when I try to debug it says document is not defined. How do I solve these two issues?

const puppeteer = require('puppeteer');

(async function scrape() {
  const browser = await puppeteer.launch({headless: false});
  const page = await browser.newPage();
  page.setDefaultNavigationTimeout(90000);
  const url = "https://excise.wb.gov.in/CHMS/Public/Page/CHMS_Public_Hospital_Bed_Availability.aspx";
  await page.goto(url, {waitUntil: 'networkidle2', timeout: 0});
  await page.waitForSelector('#ctl00_ContentPlaceHolder1_ddl_District');
  await page.select('#ctl00_ContentPlaceHolder1_ddl_District', '020');
  await page.waitForNavigation();
  let beds = await page.evaluate(() => {
    let dataRows = document.body.querySelectorAll("tbody tr");
    console.log("Num entires == " + dataRows.length);
  });
  await browser.close();
})();
Vipin Verma
  • 5,330
  • 11
  • 50
  • 92
  • Does `console.log` work inside the page.eval? – Vipin Verma May 16 '21 at 23:59
  • 1
    It logs inside the browser console, but not in your terminal – blex May 17 '21 at 00:07
  • if it logs in browser console, that is helpful too. but how do I debug the code, as printing logs does not exactly help in debugging everytime. – Vipin Verma May 17 '21 at 00:18
  • 1
    You can use the browser's debugger. Add the option to open the devtools automatically when launching the browser: `puppeteer.launch({headless: false, args: ['--auto-open-devtools-for-tabs']})` and inside your `page.evaluate`, add a `debugger;` statement where you want your code to pause, so you can inspect the variables https://imgur.com/a/dBthUcU – blex May 17 '21 at 00:22

1 Answers1

2

Selecting a city does not cause the URL to change, which is what page.waitForNavigation() waits for.

This resolves when the page navigates to a new URL or reloads.

It never happens, so your code does not continue.

You might be looking for page.waitForSelector() instead:

  // ...
  await page.select('#ctl00_ContentPlaceHolder1_ddl_District', '020');
  await page.waitForSelector('tbody tr');
  let beds = await page.evaluate(() => {
    let dataRows = document.body.querySelectorAll('tbody tr');
    return [...dataRows].map(row => row.querySelector('h5').textContent);
  });
  console.log(beds);
  await browser.close();
blex
  • 24,941
  • 5
  • 39
  • 72
  • Does `console.log` not work inside the page.eval? How do I debug the statements inside it? – Vipin Verma May 17 '21 at 00:11
  • 2
    @vipin8169 try [How do print the console output of the page in puppeter as it would appear in the browser?](https://stackoverflow.com/a/60075804/6243352) to capture logs and display them in Node. – ggorlen May 17 '21 at 00:15