Access variables with puppeteer after clicking popup or frame creating button

Question

After programming for a long time I am now trying my to get practical with Javascript. I want to load a page, and click a button and then check the available variables to test if they are looking good.

I was able to do this succesfully for just a normal page visit, however after simulating a button click that generates a popup I am not able to get the variable anymore.

Here are the steps that I want to take:

1. Open the page, accept cookies
2. Check the dataLayer
3. Click a button to add a product in a basket (generates some kind of popup/something over the normal page)
4. Check the dataLayer again

Step 2 works, but step 4 returns Undefined. I suspect it may be because it is focusing on the wrong thing, but I did not find a way to resolve this.

Here is the minimal code that reproduces the problem, I swapped out the link with example.com to be safe.

// Loading dependencies
const puppeteer = require('puppeteer');

// Opening the browser
(async () => {
  const browser = await puppeteer.launch({
    headless: false,
    defaultViewport: false
  })

// Simulating the user behavior
  const page = await browser.newPage()
  await page.goto('https://www.example.com/')
  await page.click('button#onetrust-accept-btn-handler')
  let dataLayer1 = await page.evaluate(() => {
    return window.dataLayer
  })
  await page.click('button.button.button--solid.button--custom.button--color-primary.add-to-cart')

  let dataLayer2 = await page.evaluate(() => {
    return window.dataLayer
  })
  
  console.log('First output')
  console.log(dataLayer1)
  console.log('Second output')
  console.log(dataLayer2)
  console.log('Finished')

})();

I am not sure if it is relevant but I am running on windows and updated everything to the latest versions.

Answer 1

The problem here is pretty simple. All data returned from evaluate callbacks needs to be serializable. It's a classic Puppeteer gotcha to assume DOM elements are serializable, but they aren't due to circular references and dependencies on native browser objects that won't work in Node, like document and window.

The tricky part is that Puppeteer fails silently when you try to return an object with a circular reference, defaulting to undefined. A minimal example is

const o = await page.evaluate(() => {
  const o = {};
  o.o = o; // circular reference
  return o;
});
console.log(o); // => undefined

It turns out that dataLayer is a Google Tag Manager (GTM)-related data structure. Before the click, there is no DOM node in the nested structure, so it serializes just fine, but after the click, a "gtm.element" key appears, pointing to an element associated with an event. This causes the second attempt at evaluate to fail to serialize for the circular reference reason described above.

A solution is to simply omit this "gtm.element" property during the serialization process:

const fs = require("fs").promises;
const puppeteer = require("puppeteer"); // ^18.0.4

let browser;
(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  const url = "https://www.visionexpress.com/sunglasses/gucci-gg-0631s-001/8056376305852";
  await page.goto(url, {waitUntil: "domcontentloaded"});
  await (await page.waitForSelector("#onetrust-accept-btn-handler")).click();
  await page.click(".add-to-cart");
  const dataLayer = await page.evaluate(`
    JSON.stringify(
      dataLayer,
      (k, v) => k === "gtm.element" ? undefined : v,
      2
    )
  `);
  await fs.writeFile("dataLayer.json", dataLayer);
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close())
;

It was mentioned in a comment by OP that they're looking for a key or value called (or with the substring) "becomingSkynet", but this property is nowhere in the 2.5 MB structure that results when you use a circular reference serializer such as the one from MDN which lets you serialize most of the properties in the DOM node tree (most of this data is meaningless). So it sounds like they'll need to check their assumptions and perhaps take other actions on the page to get that property to appear.