Iterate over links in a page and click based on condition

Question

I'm scraping a web page and I need to download only files on that page that meet certain criteria. How can I achieve that in puppeteer?

I'm able to locate the element using a selector and grab the attributes I need using page.$$eval, but I'm unable to figure out how to click that link.

const sectionLinks = await page.$$eval('#mainsection a', aTags => aTags.map(a => a.innerText));
  for (const sectionLink of sectionLinks) {
    if (sectionLink.toUpperCase() == 'THEONEIWANT') {
      console.log('download');
      //this is where I want to click the link
    }
  }

score 1 · Accepted Answer · answered May 22 '19 at 16:47

You are not getting the element handles. You are only returning their innerText value.

What you can do is, first get all elements and then iterate over them like this:

const elements = await page.$$('#mainsection a');
for (const el of elements) {
    const innerText = await page.evaluate(el => el.innerText, el);
    if (innerText.toUpperCase() == 'THEONEIWANT') {
        await el.click();
    }
}

This will go over all elements one by one, read their innerText value, check if the condition matches and then click it.

Optimization

This might take some time if there are a lot of links. You could improve this code by using a selector that matches based on the text you are looking for (check out this question for more information) or by using an expression like the following to check if the condition matches on the client side. This will do the check for all elements at once:

const shouldElementBeClicked = page.evaluate((...args) => args.map(el => el.innerText === '...'), ...elements);

This will result in an array with a boolean value signaling whether the element at the same position in the elements array satisfies the condition.

Iterate over links in a page and click based on condition

1 Answers1