1

I'm currently battling with a bit of a puppeteer issue I haven't faced before. I'm trying to gather urls for an individual products variants, e.g mywebsite.com/item/an-item?variant=1234.

My approach is to load up mywebsite.com/item/an-item in puppeteer, figure out how many variants there are, then loop and click on each variant (an input) then push the url to an array, and continue. This has worked for all of the other websites I've used, but not this one for some reason.

Here is my code, I've tried to both wait for the selector as well as just go for the click.

Just clicking on the selector, I get Error: No element found for selector: .swatch.selectors > div > input:nth-child(1) and when I wait for the selector, it times out.

const getVariantUrls = async (url) => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto(url);

  const variantUrls = [];
  const variantListLength = (await page.$$(".swatch.selectors > div > input"))
    .length;

  for (let i = 0; i < variantListLength; i++) {
    await page.click(`.swatch.selectors > div > input:nth-child(${i + 1})`);

    // also tried the following
    /** Errors out with:  Waiting for selector `.swatch.selectors > div > input:nth-child(1)` failed: Waiting failed: 30000ms exceeded */
    // const radio = await page.waitForSelector(
    //   `.swatch.selectors > div > input:nth-child(${i + 1})`
    // );
    // await radio.click();

    await sleep(0.5);
    const vurl = await page.url();
    variantUrls.push(vurl);
  }

  await browser.close();
  return variantUrls;
};

I don't quite understand how it isnt finding a valid selector, given I'm using nth-child with the indices that puppeteer has already told me exist. If I log variantListLength it's giving me 6 selectors, which is exactly what I expect. But then when I use the same selector with nth-child(1-6) it errors every time.

Ouroborus
  • 16,237
  • 4
  • 39
  • 62
TomLV
  • 365
  • 1
  • 11
  • I don't have your site, so it's hard to help, but instead of doing a query just to get the length, then re-querying each `nth-child`, it's simpler to store the query results and loop over them, clicking each one: `for (const el of await page.$$(".swatch.selectors > div > input")) { await el.click()}`. If visibility is a problem, you can try the native JS click: `await page.$$eval("...", els => els.forEach(e => e.click()))`. Also, [sleeping is not a good idea](https://stackoverflow.com/questions/46919013/puppeteer-wait-n-seconds-before-continuing-to-the-next-line/73676564#73676564). – ggorlen Mar 25 '23 at 16:13
  • See [this post](https://serpapi.com/blog/puppeteer-antipatterns/#assuming-puppeteers-api-works-like-the-native-browser-api) for differences between the clicks. – ggorlen Mar 25 '23 at 16:16

1 Answers1

0

Found the issue. I was debugging by running a non-headless puppeteer browser, and noticed if I scrolled down to the selector this was working, so it looks like the site requires you to visibly see the selectors to click them. Simply made the default viewport larger so the selectors were in frame on load and it solved it.

const browser = await puppeteer.launch({
    args: [`--window-size=1920,1080`],
    defaultViewport: {
      width: 1920,
      height: 1080,
    },
  });
TomLV
  • 365
  • 1
  • 11