0

I have a small java script on node.js that uses Puppeteer to scrape data from 2 websites. The loop runs indefinitely, every 10 seconds. I work in headless=false to see how the web pages behave.

When opening, a pop up/modal warning "gambling sites forbidden to children" opens. I noticed that the pop up/modal opens a few milliseconds after the page is loaded (an inertia/latency exit). The thing is that this window seems to block the code and it can't find the ".sc-jQHtVU.sc-bWXABl.gOFGcM.cHHGHO" selector (which is actually on the page)....

On the other hand, when I manually close the pop (by clicking on the "refuse" button with my cursor), the code finds the selector (and this, even if I only close one pop up on the 2 pages.

So I modified my code so that it automatically closes the pop up. The pop up closes correctly but the code returns the same error... I don't understand anything, I don't know how to solve this problem.

Here's my code, if someone can help me please.

PS: too see the pop up you can open the URL in a privacy tab

const puppeteer = require("puppeteer");
const urls = [
  "https://www.winamax.fr/paris-sportifs/match/34169987",
  "https://www.winamax.fr/paris-sportifs/match/39073593",
];
const tableau_valeurs = [];

(async () => {
  const browser = await puppeteer.launch({
    executablePath:
      "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
    headless: false,
    args: [
      "--incognito",
      "--blink-settings=imagesEnabled=false",
      "--disable-css-animations",
    ],
    defaultViewport: null,
  });
  const pages = await Promise.all(
    urls.map(async url => {
      const page = await browser.newPage();
      await page.goto(url, {
        /*waitUntil: 'networkidle0', */ timeout: 0,
        encoding: "utf-8",
      });
      return page;
    })
  );
  // wait time to let the page to fully load
  await new Promise(resolve => setTimeout(resolve, 10000));
  // method to close the pop up
  /*
  for (const page of pages) {
    await page.waitForSelector(
      ".tarteaucitronCTAButton.tarteaucitronDeny, .sc-PJClH.sc-jfdOKL.sc-fmGnzW.sc-eoXOpV.sc-kMizLa HcSjC.euooKa.dWNwst.dtmNbI.casXlC"
    );
    await page.click(
      ".tarteaucitronCTAButton.tarteaucitronDeny, .sc-PJClH.sc-jfdOKL.sc-fmGnzW.sc-eoXOpV.sc-kMizLa HcSjC.euooKa.dWNwst.dtmNbI.casXlC"
    );
  }
  // wait time to let the pop up close
  await new Promise(resolve => setTimeout(resolve, 6000));
  */
  while (true) {
    await Promise.all(
      pages.map(async page => {
        let equipeA = await page.$eval(
          ".sc-jQHtVU.sc-bWXABl.gOFGcM.cHHGHO",
          el => el.textContent.trim()
        );
        const now = new Date();
        tableau_valeurs.push(`${now}/${equipeA}`); // forme le tableau qui contient toutes les chaines de caractères scrappées
        console.log(
          `Dernier élément ajouté au tableau de valeurs: ${
            tableau_valeurs[tableau_valeurs.length - 1]
          }`
        );
        tableau_valeurs.length = 0;
      })
    );
    await new Promise(resolve => setTimeout(resolve, 10000));
  }
  await browser.close();
})();

Closing the pop up automatically doesn't work.

BadFDlower
  • 9
  • 2
  • Some of the code here doesn't seem pertinent to the problem you're asking about. To improve the question and make it easier to answer, I recommend removing the cruft, like CSV writing and so forth. Thanks. Beyond that, I recommend [not sleeping](https://stackoverflow.com/questions/46919013/puppeteer-wait-n-seconds-before-continuing-to-the-next-line/73676564#73676564) and [using non-infinite timeouts](https://blog.appsignal.com/2023/02/08/puppeteer-in-nodejs-common-mistakes-to-avoid.html#using-infinite-timeouts). – ggorlen Mar 03 '23 at 23:03
  • You can make it work with headless: true. It seems that it requires some sort of human interaction on both pages for it to work in headless: false (even clicking on the tab is enough) – AlanOnym Mar 03 '23 at 23:46
  • And there's one thing I don't understand. Even with the pop up displayed, when I inspect the element of the WEB page, the data I want to scrape are present in the HTML code. So my script should work no matter if there is a pop up or not? – BadFDlower Mar 06 '23 at 11:58

0 Answers0