Puppeteer: Click within page.evaluate() not working - but in browser

Question

In the following code snippet, I try to click a button (after some Timeout) within the page.evaluate function. It does not work. Yet, when I open the console in the launched browser and manually type const btn = document.querySelectorAll("form button")[1]; btn.click() it does.

Can anyone explain to me the cause of this difference in behavior and how to fix it?

Here's a minimal reproducible example:

import { resolve } from 'path';
import puppeteer from 'puppeteer'

//go to page and handle cookie requests
const browser = await puppeteer.launch({defaultViewport: {width: 1920, height: 1080},
        headless:false, args: ['--start-maximized']});
const page = await browser.newPage();
const url = "https://de.finance.yahoo.com/";
await page.goto(url);
await page.waitForSelector("div.actions");
await page.evaluate( () => {
let z= document.querySelector("div.actions"); z.children[4].click()
})

await page.waitForSelector("input[id=yfin-usr-qry]");
await page.evaluate( () => {let z= document.querySelector("input[id=yfin-usr-qry]");
    z.value = "AAPL"; const btn = document.querySelectorAll("form button")[1]; 
    return new Promise((resolve) => setTimeout(() => {btn.click();resolve()},1000))})
})

Please share a [mcve] with the actual page and your complete script. There are many reasons why something that works in the browser doesn't work in Puppeteer, ranging from bot detection blocks to iframes to shadow roots to unexpected dynamic JS behavior to visibility issues. Also, [timeouts in any form are bad practice and you probably don't need them](https://serpapi.com/blog/puppeteer-antipatterns/#overusing-waitfortimeout). Even if you do, the code waits for the timeout after selecting the button, not before. Usually, the first step is to use `waitForSelector` if the element is dynamic. — ggorlen, Sep 28 '22 at 15:19
Thank you for the clarification. I have now updated the post to include a minimal working example. — P.Jo, Sep 30 '22 at 12:56
Maybe the button doesn't exist yet? You wait for some input and you have a 1s delay but your delay is after you already queried the button. — CherryDT, Sep 30 '22 at 15:20
True, but if I remember correctly I also had a version with a console.log(btn) - showing that it exists. Hence, there needs to be a different cause of this problem. (Even introducing more delay in the TimeOut ends up with the same result - the button is clicked - but the website does not respond to it) — P.Jo, Oct 01 '22 at 15:04

ggorlen · Accepted Answer · 2022-10-01T15:13:19.513

The form button selector appears to be incorrect, selecting a non-visible element with class .modules_clearBtn__uUU5h.modules_noDisplay__Qnbur. I'd suggest selecting by .finsrch-btn or #UH-0-UH-0-Header .finsrch-btn if you have to select this, but it's not really necessary, so I won't use it in my suggested solution below.

Beyond that, I'd tighten up some of the selectors, skip the timeout and prefer using trusted Puppeteer events when possible.

I'm not sure what data you want on the final page but this should give you a screenshot of it, showing all of the content:

const puppeteer = require("puppeteer"); // ^18.0.4

let browser;
(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  const $ = (...args) => page.waitForSelector(...args);
  const url = "https://de.finance.yahoo.com/";
  await page.goto(url, {waitUntil: "domcontentloaded"});
  await (await $('button[name="agree"]')).click();
  const input = await $("#yfin-usr-qry");
  await input.type("AAPL");
  await page.keyboard.press("Enter");
  await $("#AAPL-interactive-2col-qsp-m");
  await page.evaluate("scrollTo(0, document.body.scrollHeight)");
  await $("#recommendations-by-symbol");
  await page.screenshot({path: "aapl.png", fullPage: true});
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

That said, rather than navigating to the homepage, typing in a search, then pressing a button, you could consider building the URL directly, e.g. https://de.finance.yahoo.com/quote/${symbol} and navigating right to it. This is generally faster, more reliable, and easier to code.

Thank's a lot. The code and comments on improvement help a lot. In fact, I can see what are bad habits (setTimeout) and how to avoid them. The suggestion on querying the URL directly is also helpful. But one last point: It would also be great if anyone could get me some hint as to why in my code the click event did not succeed. This would give me some new insights! — P.Jo, Oct 01 '22 at 15:00
As for the click, the `form button` selector appears to be incorrect, selecting a non-visible element with class `.modules_clearBtn__uUU5h.modules_noDisplay__Qnbur`. Take a look in the element inspector. I'd suggest selecting by `.finsrch-btn` or `#UH-0-UH-0-Header .finsrch-btn`. As mentioned in the post, it's odd to select, then wait a second before clicking. Usually, it's the other way around: wait, then select and click as soon as you get the element. But it appears to be there in the static HTML so waiting isn't necessary. Pressing Enter (or navigating) circumvents the problem entirely. — ggorlen, Oct 01 '22 at 15:05

Puppeteer: Click within page.evaluate() not working - but in browser

1 Answers1