Unable to locate elements on webpage with headless chrome

Question

I have a script that's accessing printers, and my code works totally fine when chrome is run normally, but when it's run headless, selenium can't seem to find elements on the webpage.

Here's the relevant code:

init method:

def __init__(self, ip_address):
    """ Initialize a new Printer_Webpage object."""
    self.ip_address = ip_address
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--window-size=1920x1080")
    self.browser = webdriver.Chrome(chrome_options=chrome_options)
    # Ignore lack of cert for each printer web page.
    # Otherwise, can't open page.
    self.browser.accept_untrusted_certs = True

Login method:

def login(self):
    """Navigates through the login page for the printer."""
    # Open login page
    self.browser.get(f'https://{self.ip_address}/wcd/top.xml')
    # STEPS TO LOGIN:
    # 1) Select 'Administrator' radio button and click.
    self.browser.find_element_by_id('Admin').click()
    # 2) Select Login button and click.
    self.browser.find_element_by_xpath("//input[@type='submit' \
                                        and @value='Login']").click()
    # 3) Select admin (user mode)
    self.browser.find_element_by_id('R_ADM2').click()
    # 4) Select password field and input PASSWORD, then submit.
    password_field = self.browser.find_element_by_id('Admin_Pass')
    password_field.send_keys(PASSWORD)
    password_field.send_keys(Keys.RETURN)

Full error message:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"Admin"}

And here's some other info that might be of use:

(Session info: headless chrome=62.0.3202.94)

(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 10.0.14393 x86_64)

have the headless browser take a screenshot. It might be a matter of either scrolling to the element first or you may need to resize the headless browser when you initialize it - if it's not big enough the site may be sized for mobile and doesn't have the admin link in that view. — chris-crush-code, Nov 15 '17 at 20:42
Use a comma for the size of the window: `--window-size=1920,1080`. Take a screenshot or dump the HTML from the page to figure out why the element is not present. — Florent B., Nov 15 '17 at 20:43
Okay so I've found that the webpage isn't loading properly, and ends up just displaying a blank white page. When loading normally, the page is white a for a few seconds, so I tried using time.sleep(30) to give it plenty of time to load, and it still shows up as a white page. — Kuba Wernerowski, Nov 15 '17 at 21:46
Chrome-headless does not work with invalid https certificates. You will only get a blank page! See https://bugs.chromium.org/p/chromium/issues/detail?id=721739 — SiKing, Nov 15 '17 at 23:38
Thanks! Is there no work around for this, or will I have to switch to a different headless browser? — Kuba Wernerowski, Nov 16 '17 at 00:29

Raj NIgam · Accepted Answer · 2020-07-24T05:28:48.297

I had the same problem. You could take screenshots to understand whats wrong.

driver.get_screenshot_as_file("screenshot.png")

A few reasons why selenium works when run normally but stops working in headless mode -

1)It might have switched to a mobile template. Can be fixed by changing the window size.

chrome_options.add_argument("--window-size=1920,1080")

2)If its a blank page (screenshot), it might be due to an invalid SSL certificate.(see @Marcel_Wilson post) It should be fixed by -

chrome_options.add_argument('--ignore-certificate-errors')
chrome_options.add_argument('--allow-running-insecure-content')

3)Its possible that the website blocks 'headless' mode. (Your screenshots might show errors which you cannot recreate in normal mode) You could try this-

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36'
options.add_argument(f'user-agent={user_agent}')

However, the above code won't work if the website has a more robust blocking method. You can find more about this here https://intoli.com/blog/making-chrome-headless-undetectable/.

Raj spot on with Number 3 for me, I posted a more in depth answer here as there were other things to look out for as the site that made some changes to block headless also did not work properly in earlier versions of Chrome: https://stackoverflow.com/a/69464060/1871891 — user1871891, Oct 06 '21 at 10:47

score 9 · Answer 2 · answered May 10 '19 at 09:19

I met the same situation. After studying, the following is correct：

self.chrome_options = webdriver.ChromeOptions()
self.chrome_options.add_argument("--window-size=1920,1080")
self.chrome_options.add_argument("--disable-extensions")
self.chrome_options.add_argument("--proxy-server='direct://'")
self.chrome_options.add_argument("--proxy-bypass-list=*")
self.chrome_options.add_argument("--start-maximized")
self.chrome_options.add_argument('--headless')
self.chrome_options.add_argument('--disable-gpu')
self.chrome_options.add_argument('--disable-dev-shm-usage')
self.chrome_options.add_argument('--no-sandbox')
self.chrome_options.add_argument('--ignore-certificate-errors')
self.browser = webdriver.Chrome(options=self.chrome_options)

score 6 · Answer 3 · answered Nov 17 '17 at 16:45

6

If it's an issue with SSL certs, you can start Chrome without certs using a command line flag (assuming that's how you're starting it). I believe the switch is --allow-running-insecure-content, and I located that from this list found here.

answered Nov 17 '17 at 16:45

browserless

2,090
16
16

score 2 · Answer 4 · answered May 03 '18 at 20:33

2

@siking pointed out in the comments...

Chrome-headless does not work with invalid https certificates. You will only get a blank page! See http://bugs.chromium.org/p/chromium/issues/detail?id=721739

answered May 03 '18 at 20:33

Marcel Wilson

3,842
1
26
55

score 2 · Answer 5 · edited Mar 21 '19 at 16:42

2

I had the same problem and found that the elements were loading slower in headless. By adding the following line of code, the problem went away:

driver.manage().timeouts().implicitlyWait(5, TimeUnit.SECONDS);

edited Mar 21 '19 at 16:42

double-beep

5,031
17
33
41

answered Mar 21 '19 at 16:21

NumberzQA

23
3

score 2 · Answer 6 · answered Sep 12 '20 at 07:19

I did face the issue but the below code worked like a champ for me.

    final ChromeOptions options = new ChromeOptions();
    options.addArguments("--window-size=1920,1080");
    options.addArguments("--allow-insecure-localhost");
    options.addArguments("--headless");
    options.addArguments("--disable-gpu");
    options.addArguments("--no-sandbox");
    DesiredCapabilities caps = DesiredCapabilities.chrome();
    caps.setCapability(ChromeOptions.CAPABILITY, options);
    caps.setCapability("acceptInsecureCerts", caps);
    WebDriver driver = new ChromeDriver(options);

score 0 · Answer 7 · answered Nov 17 '21 at 11:55

I had the same issue that headless doesn't work with some Websites and Elements (showing blank page content, not finding an element etc.).

This is most likely because of a missing User-Agent or a small window size. Add the following Arguments:

"--user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36"
"--window-size=1920x1080"

In case the above doesn't work, there is also another method to "simulate" the headless mode by minimizing and moving the window to a location that can not be seen by the user.

This doesn't hide the chrome task from the taskbar, but the Chrome tab itself will still be hidden for the User.

Just use the following arguments:

var chromeOptions = new ChromeOptions();
chromeOptions.AddArguments(new List<string>() { "--window-size=1,1", "window-position=-2000,0" });  // This hides the chrome window

var chromeDriverService = ChromeDriverService.CreateDefaultService();
chromeDriverService.HideCommandPromptWindow = true;    // This is to hid the console.

ChromeDriver driver = new ChromeDriver(chromeDriverService, chromeOptions);
//driver.Manage().Window.Minimize(); //use this if the code above does not work
driver.Navigate().GoToUrl("https://google.com");

score 0 · Answer 8 · answered Dec 14 '21 at 12:29

time.sleep(0.2) fixed it for me.

I had the same issue. In headless mode one webElement could not be found and selenium was throwing the NoSuchElementException with ERROR: Message: no such element: Unable to locate element.

What fixed it for me was adding a wait time of 0.2 seconds with time.sleep(0.2) the line above locating this webElement and that fixed it for me.

For some reason it needed a little more time to load this element in headless mode.

Unable to locate elements on webpage with headless chrome

8 Answers8

Linked