0

I have some URL parameters that I need to pass to Puppeteer via query strings. Everything works fine unless I define the necessary variable before I call the .evaluate() function. For instance, if I get the parameters of a URL, create a variable for a specific parameter, run .evaluate(), and try to access the variable within that it comes back undefined.

Here is an example

const testurl = 'https://example.com/app.js?&url=https://example.com/pagetocrawl.html&elements=.links'
const params = new URLSearchParams(testurl);
const url = params.get('url');
const elements = params.get('elements');
const networkidle = 'networkidle2';
const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto(url, {waitUntil: networkidle});
pageitems = await page.evaluate(() => {
let results;
let items = document.querySelectorAll(elements);
items.forEach((item) => {
results += item.innerHTML;
});
return results;
});

The above code always returns an error saying that the elements variable is undefined. If I try to declare it inside the evaluate() call then it says that params is undefined, so unless I know what element to look for without having to get that from the URL I can't crawl the page. I tried changing evaluate(()=> to evaluate((params) => and then declaring the elements variable but that caused an error saying "Cannot read property 'get' of undefined." How do I define the variables outside the evaluate() function and be able to access them inside of it? Right now the next thing I can think of would be to make another call to get the current URL inside the evaluate() function, create a new params variable based on that, and create a new elements variable that gets its value from the new params variable.

PostAlmostAnything
  • 105
  • 2
  • 11

1 Answers1

1

You just need to provide the value of the variable as an argument of evaluate() (see page.evaluate()):

pageitems = await page.evaluate((selector) => {
  let results;
  let items = document.querySelectorAll(selector);
  items.forEach((item) => {
    results += item.innerHTML;
  });
  return results;
}, elements);
vsemozhebuty
  • 12,992
  • 1
  • 26
  • 26
  • I found something like that works for variable, but now i need to pass a function also. I have a function called getContent() which I need to pass parameters to from a foreach loop in evaluate() but if I try to pass the function using that syntax ex: page.evaluae((getContent()) => it does not work. How do I make a function accessible within that? – PostAlmostAnything Mar 01 '21 at 01:01
  • Unfortunately, you can transfer only serializable values as arguments, and functions are not serializable. So you need to make this function a part of the code of the main function in `evaluate()`. – vsemozhebuty Mar 01 '21 at 06:53