4

I wonder whether or not exists some flag or tag that the website can use it to detect the request came from Puppeteer?

When I ran my code based on Puppeteer to visit the target website, I found that the website seems to know the request was made by Puppeteer.

How can it do?

Nicolò Gasparini
  • 2,228
  • 2
  • 24
  • 53
kazaff
  • 137
  • 1
  • 6
  • 1
    I think it may be because of Chrome setup some header in the request when it runs under "automated test software" mode. But I can't find that “header” in Dev-tools or fiddler~ And in order to test, I run the request by the same chromium manually, Everything is OK, The website does what I expected. – kazaff Oct 26 '19 at 03:12

2 Answers2

4

If you are running the puppeteer and would like to pass some information to the website to catch your crawling, the best way to do so would be to set a custom user agent:

const browser = await puppeteer.launch({
    args: ['--user-agent=hhh'],
});
const page = await browser.newPage();

See here more info

Viceversa, if you own a website and would like to know if the visits are real or from a bot (puppeteer, a scraper, or anything else) see this answer for some of them. Also this answer

Nicolò Gasparini
  • 2,228
  • 2
  • 24
  • 53
3

I found a way to cross the limitation. It's an easy way:

const browser = await puppeteer.launch({headless: false, ignoreDefaultArgs: ["--enable-automation"],});

This will let the browser to not setup navigator.webdriver variable.

kazaff
  • 137
  • 1
  • 6