0

I am running Selenium headless on a Linux machine, and it can't find a specific element. However, I run the same exact code on a Windows machine and it finds the element without a problem. I have the latest version of the driver on my Linux machine. Any ideas how to fix this?

options = Options()
options.add_argument('--no-sandbox')
options.add_argument("--headless")
options.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                     'Chrome/97.0.4692.71 Safari/537.36')
with webdriver.Chrome(options=options) as driver:
    driver.get(url)
    driver.find_element(By.ID, "btnhome").click()
    video = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.TAG_NAME, "video"))).get_attribute("src")

The problem element is the video element. I have increased the wait time to 2 minutes as well, just for debugging. It works fine on Windows headless, but can't find it on Linux headless.

Thanks

Michael Mintz
  • 9,007
  • 6
  • 31
  • 48
goalie1998
  • 1,427
  • 1
  • 9
  • 16
  • One quick thing you can do to check is to disable headless, let the browser be visible and and see where the issue is occurring. – Anand Gautam Jan 16 '22 at 09:52
  • 1
    Instead of `presence_of_element_located` can you try with `element_to_be_clickable` or `visibility_of_element_located` and update us if that helps ? – cruisepandey Jan 16 '22 at 09:52
  • I second @curisepandey. presence of element gets a green even if the element is in background and not necessarily get visible and clickable in DOM. – Anand Gautam Jan 16 '22 at 09:53
  • @AnandGautam Unfortunately the Linux system is a non-GUI system, so I can't run it unless it is headless – goalie1998 Jan 16 '22 at 10:06
  • @cruisepandey Nope, neither worked – goalie1998 Jan 16 '22 at 10:09
  • Take screenshot or print page source after `driver.get(url)` to be sure, you've not redirected to some other page, e.g. detected as a bot or got some security alert. – Max Daroshchanka Jan 16 '22 at 10:44
  • 1
    Using https://stackoverflow.com/a/6300672/7058266 should help you get around your issue. I put all the details you need in the longer answer I posted. – Michael Mintz Jan 16 '22 at 15:46

1 Answers1

1

2023 Update: I'm revising my original answer to use https://stackoverflow.com/a/73840130/7058266 as a better solution. There's a new Chrome headless mode that gets the same results as regular Chrome:

options.add_argument("--headless=new")

The solution for this involves using https://github.com/ponty/PyVirtualDisplay in order to simulate a headed display in a headless environment such as Linux. (This can help you get around issues that might only appear when loading a website with a headless browser.) The virtual display uses Xvfb, and there's actually a very popular Stack Overflow solution here about it: https://stackoverflow.com/a/6300672

Here's a summary of the previous solution for your situation:

To install:

pip install pyvirtualdisplay

And here's how to use it in your Python code:

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
# display is active / Run your selenium code
# ----------
display.stop()
# display is stopped

After putting it all together with your code, it will look something like this:

from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
# ...
options = Options()
options.add_argument('--no-sandbox')
options.add_argument('user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) '
                     'Chrome/97.0.4692.71 Safari/537.36')
with webdriver.Chrome(options=options) as driver:
    driver.get(url)
    driver.find_element(By.ID, "btnhome").click()
    video = WebDriverWait(driver, 10).until(
            EC.presence_of_element_located((By.TAG_NAME, "video"))).get_attribute("src")
# ...
display.stop()

(Make modifications as necessary, such as the screen resolution, etc.)

There are some Selenium Python frameworks that already have the virtual display built-in (such as SeleniumBase, which activates the virtual display via an added pytest command-line arg: --xvfb.) Each framework that has the feature built-in may do things a bit differently.

Michael Mintz
  • 9,007
  • 6
  • 31
  • 48