0

I have to scrape several websites to get some data. I'm using Selenium ChromeDriver with C#. I initialized my ChromeDriver in this way:

var options = new ChromeOptions();
options.AddArguments("headless");
options.AddArgument("--log-level=3");
options.AddArgument("--no-sandbox");
options.AddArgument("--silent");
options.AddArgument("--user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36");
string pathChromeDriver = Directory.GetCurrentDirectory(); 
driver = new ChromeDriver(pathChromeDriver,options,TimeSpan.FromMinutes(3));

The problem is that if I navigate to the url of the page with driver.Navigate().GoToUrl(url);, the driver doesn't go to the page from which I have to read data, but in the resulting page there is a "I'm not a robot" captcha. If I open the page manually from a browser, it opens the page normally. Is there a way to scrape the page appearing as a human user? Thanks

Alfredo Torre
  • 678
  • 1
  • 9
  • 25
  • Did You try in incognito mode? Or to test with Your profile included in webDriver – Kovacic Jul 20 '18 at 08:16
  • I tried including my profile with this: `driverOptions.AddArguments($"user-data-dir=C:/Users/{Environment.UserName}/AppData/Local/Google/Chrome/User Data/Default");` or in incognito mode, but with same results – Alfredo Torre Jul 20 '18 at 08:31
  • I think that captcha was invented exactly for those purposes - to prevent non-user activity :) – Yeldar Kurmangaliyev Jul 20 '18 at 09:19
  • By the way, this article may be useful for you: https://stackoverflow.com/questions/33225947/can-a-website-detect-when-you-are-using-selenium-with-chromedriver – Yeldar Kurmangaliyev Jul 20 '18 at 09:21

0 Answers0