Selenium Python Chrome: Last element in drop down list is not visible in Headless mode

Question

I want to use Selenium to click an element in a menu drop down list. Unfortunately the element I want to click is all the way at the bottom of the list. My Selenium Python Chrome code runs perfectly in foreground mode, but when I run it in background / headless mode I get a timeout, or not clickable error (see notes below). Here is the code, using the actual public url, so you can test it out if you want:

# define chrome options
options = webdriver.ChromeOptions()
# make sure browser is maximized so all elements are visible
options.add_argument('--start-maximized')
# define driver
driver = webdriver.Chrome(chrome_options=options)
# define function used to get button
def getbutton(driver, xpath, waittime=10):
    return WebDriverWait(driver, waittime).until(EC.element_to_be_clickable((By.XPATH, xpath)))
# navigate to url
driver.get('http://oasis.caiso.com/mrioasis/logon.do')
# another two different ways to make sure browser window is maximized
# I'm going through this trouble just in case selenium needs the window
# maximized to see the element
driver.fullscreen_window()
driver.maximize_window() 
# choose dropdown menu
button1 = getbutton(driver, "//span[contains(@id, 'LBL_')][text()='ATLAS REFERENCE']", waittime=120)
hover = ActionChains(driver).move_to_element(button1)
hover.perform()
time.sleep(5)
# see about navigating to the second to last button in the drop down 
# menu list - not what I want, but a test
testing = True
if testing:
    # move to second to last button - interestingly this works in foregraound AND background / headless modes
    button1a = getbutton(driver, "//span[contains(@id, 'LBL_')][text()='Intertie Scheduling Limit and Tie Mapping']")
    hover = ActionChains(driver).move_to_element(button1a)
    # THIS WORKS in foreground AND backgroubd modes
    hover.perform()
    time.sleep(10)
# Now try and click the button I really want - the LAST button in the drop down menu list
button2 = getbutton(driver, "//span[contains(@id, 'LBL_')][text()='Master Control Area Generating Capability List']", waittime=120)
button2.click()

button2 = getbutton fails with the following error message:

selenium.common.exceptions.TimeoutException

Interestingly, if I turn testing to False, I get the following error message:

selenium.common.exceptions.WebDriverException: Message: unknown error:
Element <span ...></table>">Master Control Area Generating Capability List</span>
is not clickable at point (128, 599). Other element would receive the click

It seems like selenium is unable to see the element in background / headless mode, even though neither of these exceptions occur when running in foreground mode.

I'm running Python in Windows Server 2012 R2 for what it's worth.

Thanks for any help!

This sounds like an [X-Y problem](http://xyproblem.info/). Instead of asking for help with your solution to the problem, edit your question and ask about the actual problem. What are you trying to do? — undetected Selenium, Feb 21 '19 at 05:08
The problem is that I want to use selenium to click the bottom element of a menu drop down list, but for some reason selenium can't see the last element. I've edited my OP to make that more clear. Thanks for the suggestion. — David Miller, Feb 21 '19 at 13:10

Bruno Robert · Answer 1 · 2019-02-21T15:32:35.227

1

I've changed a few things in your code and I no longer get the errors. I'm not sure if it's loading the correct webpage though since it's headless and everything seems to be dynamically loaded.

# another two different ways to make sure browser window is maximized
# I'm going through this trouble just in case selenium needs the window
# maximized to see the element
#driver.fullscreen_window()
#driver.maximize_window() 
# choose dropdown menu
button1 = driver.find_element_by_xpath("//span[contains(@id, 'LBL_')][text()='ATLAS REFERENCE']")
hover = ActionChains(driver)
hover.move_to_element(button1)
# see about navigating to the second to last button in the drop down 
# menu list - not what I want, but a test
testing = True
if testing:
    # move to second to last button - interestingly this works in foregraound AND background / headless modes
    button1a = driver.find_element_by_xpath("//span[contains(@id, 'LBL_')][text()='Intertie Scheduling Limit and Tie Mapping']")
    hover.move_to_element(button1a)
    # THIS WORKS in foreground AND backgroubd modes
# Now try and click the button I really want - the LAST button in the drop down menu list
button2 = driver.find_element_by_xpath("//span[contains(@id, 'LBL_')][text()='Master Control Area Generating Capability List']")
hover.click(button2)
hover.perform()

I'm selecting the elements with .find_element_by_xpath() and I'm queuing up all the ActionChains to execute them all at the end with hover.perform().

After further analysis, I'm unable to locate the "masterControlAreaGenCapGrid_GRID_LABEL" element with driver.find_element_by_id("masterControlAreaGenCapGrid_GRID_LABEL").text It works when the browser isn't headless though...

edited Feb 21 '19 at 15:32

answered Feb 21 '19 at 15:23

Bruno Robert

302
2
8

Thanks Robert for checking out my issue, but it sounds like you're having exactly the same problem, namely that it works in non headless mode, but not in headless mode. At least you've reproduced the problem I'm having. I wonder how to solve it. Any other ideas? – David Miller Feb 22 '19 at 16:34
1

If I were to guess, it's a rendering issue, chromedriver must render the page differently when headless and thus some things don't work exactly (probably some javascript isn't getting executed) the same as when it's GUI is being used. You could try using firefox instead of chrome to see if there is a difference. – Bruno Robert Feb 22 '19 at 20:26
1

You could try using xvfb with pyvirtualdisplay like in this [discussion](https://stackoverflow.com/questions/45370018/selenium-working-with-chrome-but-not-headless-chrome?rq=1) . xvfb stands for "X virtual frame buffer", my understanding is that it loads stuff in memory as if it was going to display it but it doesn't actually display it on screen. – Bruno Robert Feb 22 '19 at 20:34
Unfortunately I'm on windows and pyvirtualdisplay does not work on windows - see [link](https://stackoverflow.com/questions/43270513/is-there-a-windows-equivalent-to-pyvirtualdisplay). I also tried another suggestion from that same discussion: options.add_argument('--window-size=1920, 1080') But that did not help either. I'm still totally stuck. – David Miller Feb 24 '19 at 19:27
I did figure out a way to print the screen even in headless mode: driver.save_screenshot(file) driver.set_window_size(1920, 1080) time.sleep(5) size = self.driver.get_window_size() print("Window size: width = {}px, height = {}px.".format(size["width"], size["height"])) – David Miller Mar 06 '19 at 14:20

score 0 · Answer 2 · answered Mar 06 '19 at 14:18

So I finally figured out what the problem is: Running Windows task scheduler with option 'run whether user is logged on or not' only opens a small browser (1024x768) that CANNOT be resized, even with all the great suggestions being offered here. So I simply cannot see the element I want to click.

See the same issue resolved here: screen resolution in mode "Run whether user is logged on or not", in windows task scheduler

So the less than ideal workaround is to only run when user is logged on.

Thanks for all your help!

Selenium Python Chrome: Last element in drop down list is not visible in Headless mode

2 Answers2