Puppeteer loads blank page with 429 when accessing URL

Question

I am working on a project to obtain pricing information from a hotel website, but I cannot perform any searches when loading the website in puppeteer.

Here is a snippet of my JavaScript that opens Chrome.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch({ headless: false, devTools: false });
  const page = await browser.newPage();
  await page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3683.103 Safari/537.36');

  await page.goto('https://www.hyatt.com/');

})();

How are they detecting that I'm using Chrome controlled by Puppeteer, even though it is running a headful browser?

Thanks

score 2 · Accepted Answer · answered Jun 08 '19 at 01:50

2

They're fingerprinting you. It's some javascript that enumerates things like your browser plugins and navigator object and sends that back to the server.

So you'd have to intercept that post and change it to something a normal browser would send.

answered Jun 08 '19 at 01:50

pguardiario

53,827
19
119
159

Can you give a detailed example? I'm struggling with this issue as well. I'm using my google chrome to access the URL, everything is fine. However, when using Puppeteer with Chromium, the URL keeps returning 429 and a blank page. – Erron Developer Aug 12 '20 at 01:00

score 1 · Answer 2 · answered Jun 07 '19 at 21:43

Try adding the following header which is expected

'accept-language' : 'en-US,en;q=0.9'

works for me in other languages.

From this answer it looks like you need something like:

await page.setExtraHTTPHeaders({
    'Accept-Language': 'en-US,en;q=0.9'
});

score 1 · Answer 3 · answered Jun 08 '19 at 16:11

The page is using Fingerprintjs2 to analyse your browser before letting you access the page.

On your first request, the page is sending you to a special detection page, which is using Fingerprint2. Quote from the github repository:

Modern & flexible browser fingerprinting library

The code will check all kinds of browser information, like specific attributes, timezone, pixelRatio of your device, etc. You can find the actual source code in this file on github.

You will have to change all kind of browser attributes to make the fingerprinting library actually think you are a "normal user".

Puppeteer loads blank page with 429 when accessing URL

3 Answers3