4

I'm trying to load a website with Chrome browser in headless mode using Selenium web driver. I face an issue with some specific websites. The page is loading, in the first 2-3 seconds it shows a page with "please enable javascript..." and after 3 seconds, page source goes blank.

I'm using Selenium and especially Chrome for long time and I am familiar with the platform. For the purpose of this case, I'm using Chrome Version 73.0.3683.86 , ChromeDriver 2.46.628411 (which is compatible according to Which ChromeDriver version is compatible with which Chrome Browser version?) on a Mac OS. selenium java version is latest - 3.141.59

I suspect that headless Chrome cannot handle specific content-type such as "svg" and any other GUI related HTTP response. 

    ChromeOptions chromeOptions = new ChromeOptions();
    chromeOptions.addArguments("--headless");
    WebDriver driver = new ChromeDriver(chromeOptions);

    driver.get("https://identity.tescobank.com/login");

    Thread.sleep(3000);

    System.out.println(driver.getPageSource());

    driver.quit();

Expected result is to have the page source same as it is showing in non-headless mode.

Adi Ohana
  • 927
  • 2
  • 13
  • 18
  • Headless chrome can do anything normal chrome can do; it just renders into an off-screen buffer. The JS warning is weird. How can that vanish when JS is disabled? My guess is that the page (login page of a bank) detects Selenium somehow and shuts down for security reasons. – Aaron Digulla Apr 02 '19 at 16:04
  • When I'm using Selenium in non-headless bank page is loading perfectly – Adi Ohana Apr 02 '19 at 16:05
  • This is very weird. Check for errors in the log. Do you see JavaScript errors (i.e. the JavaScript console) in headless mode somewhere? Did you try to other web pages like google.com? – Aaron Digulla Apr 02 '19 at 16:09
  • I'm talking about specific websites, of course google is working. Chrome driver verbose logging is not showing any errors. What I do see is some svg and video files loading and I suspect that this is the reason. – Adi Ohana Apr 02 '19 at 16:12
  • SVG should work. Video and Flash probably won't. – Aaron Digulla Apr 02 '19 at 16:19
  • so what you suggest? how we workaround this? – Adi Ohana Apr 02 '19 at 16:35
  • You say "Specific webistes" is it possible the website to check the agent, and have some logic that implements displaying such error for unknown browsers or something like that? Can you give example of such public website? – Jeni Apr 02 '19 at 17:33
  • How the website could tell if I’m using Selenium? I the example above there is an example.. – Adi Ohana Apr 02 '19 at 17:35
  • Websites like to look at the user agent string. Also it might be possible to detect Selenium / headless chrome using JavaScript. – Aaron Digulla Apr 03 '19 at 11:21
  • thanks @AaronDigulla. How is it possible to detect HEADLESS chrome? what is the difference between headless and non-headless? – Adi Ohana Apr 03 '19 at 11:42
  • 1
    I have no idea. I'd start with using the DevTools remotely as described here: https://developers.google.com/web/updates/2017/04/headless-chrome I did that and I see `HeadlessChrome` in the user agent. – Aaron Digulla Apr 03 '19 at 12:31

2 Answers2

4

Headless Chrome should be able to handle everything the normal Chrome can do:

It brings all modern web platform features provided by Chromium and the Blink rendering engine to the command line.

(see https://developers.google.com/web/updates/2017/04/headless-chrome)

Since only the login page of a bank causes you trouble, my guess is that the security of the page detects an anomaly and decides not to serve you.

One way they can do that is by looking at the User Agent string which contains HeadlessChrome.

That said, unless you're writing integration tests for the bank, your behavior is at least suspicious. If you have a valid and legal concern, clear it with the bank first. They might take actions against you, otherwise. Blocking your IP address (which could affect many people) or asking the police to have a word with you.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
  • Thanks ! overriding the HeadlessChrome in user agent solved the issue – Adi Ohana Apr 03 '19 at 14:31
  • how did you do that? @AdiOhana – Noel Omondi Aug 11 '20 at 16:56
  • @NoelOmondi chromeOptions.addArguments("--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"); – Adi Ohana Aug 11 '20 at 17:23
  • Thanks this solved it for me! Although I needed to do more things in my case as can be seen by more response here: https://stackoverflow.com/a/69464060/1871891 – user1871891 Oct 06 '21 at 11:10
2

I was facing similar issue in my script, after login. Somehow refreshing the page resolved the issue. driver.navigate().refresh();

Shubham Jain
  • 21
  • 1