The solution for this involves using https://github.com/ponty/PyVirtualDisplay in order to simulate a headed display in a headless environment such as Linux. (This can help you get around issues that might only appear when loading a website with a headless browser.) The virtual display uses Xvfb, and there's actually a very popular Stack Overflow solution here about it: https://stackoverflow.com/a/6300672
Here's a summary of the previous solution for your situation:
To install:
pip install pyvirtualdisplay
And here's how to use it in your Python code:
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
# display is active / Run your selenium code
# ----------
display.stop()
# display is stopped
After putting it all together with your code, it may look something like this:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
# ...
option = webdriver.ChromeOptions()
driver = webdriver.Chrome(ChromeDriverManager().install(), options=option)
# ...
display.stop()
(Make modifications as necessary, such as the screen resolution, etc.)
There are some Selenium Python frameworks that already have the virtual display built-in (such as SeleniumBase, which activates the virtual display via an added pytest command-line arg: --xvfb.) Each framework that has the feature built-in may do things a bit differently.
Just be sure NOT to run Chrome in headless mode if you use this solution. The virtual display should allow you to run a regular headed Chrome in a headless environment using Xvfb.
