How to retrieve reCAPTCHA token from iframe with Puppeteer?

Question

I'm writing a bot with which I am trying to scrape a reCAPTCHA token after a task has been completed. I'm trying to use:

await page.evaluate(() => document.getElementById('recaptcha-token').value)

after the captcha has loaded onto the page, however each time I've been getting the same error: Uncaught (in promise) Error: Evaluation failed: TypeError: Cannot read property 'value' of null.

I believe that this error is in part caused by the fact that the element that I'm trying to fetch is of type hidden:

<input type="hidden" id="recaptcha-token value="[very long string of letters and numbers]">

How would I go about bypassing this?

Answer 1

Firstly I really recommend you to read Thomas Dondorf's answer on Puppeteer + reCAPTCHA topic.

If you are still willing to go this way then read my answer below:

---

The fact the <input> is type=hidden is not affecting how puppeteer interacts with the element as it is already in the DOM. You can even test it from Chrome DevTools Console tab by running $('#recaptcha-token').value: you will get its value without any problems. In fact the problem lies somewhere else.

Your are facing currently two issues:

1.) The reCAPTCHA is inside an iframe you need to step inside to let Puppeteer interact with the desired element. To achieve this you will need to grab the exact iframe by its element handle, then using contentFrame() to switch from "browser" to "frame" context.

2.) You will also need the following security disabling args to launch puppeteer: args: ['--disable-web-security', '--disable-features=IsolateOrigins,site-per-process'] because due to the same-origin policy you are not allowed to go inside the iframe by default.

Example reCAPTCHA page: https://patrickhlauke.github.io/recaptcha/

Example script:

const puppeteer = require('puppeteer')

async function getToken() {
  const browser = await puppeteer.launch({
    headless: false,
    args: ['--disable-web-security', '--disable-features=IsolateOrigins,site-per-process']
  })
  const page = await browser.newPage()

  try {
    await page.goto('https://patrickhlauke.github.io/recaptcha/')

    await page.waitForSelector('.g-recaptcha > div > div > iframe')
    const elementHandle = await page.$('.g-recaptcha > div > div > iframe')
    const frame = await elementHandle.contentFrame()
    const value = await frame.evaluate(() => document.getElementById('recaptcha-token').value)
    console.log(value)
  } catch (e) {
    console.error(e)
  }

  await browser.close()
}
getToken()