4

I try to run chromedriver via selenium in headless mode.

IMPORTANT The code runs perfectly fine if I eliminate the following code lines (but is not headless):

chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')

This is the error I get when I try to implement the headless argument:

Traceback (most recent call last):
  File "camel.py", line 83, in <module>
    executable_path=executable_path)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/.local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: failed to wait for extension background page to load: chrome-extension://jkompbllimaoekaogchhkmkdogpkhojg/_generated_background_page.html
from unknown error: page could not be found: chrome-extension://jkompbllimaoekaogchhkmkdogpkhojg/_generated_background_page.html

This are the lines 81, 82 and 83

chrome_options.add_extension(extension_path)
driver = webdriver.Chrome(options=chrome_options,
                          executable_path=executable_path)

This is the code (the crhomedriver execution parts):

from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.expected_conditions import presence_of_element_located

import os
BASE_DIR = os.path.dirname(os.path.abspath(__file__))

chrome_options = Options()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
log_path = os.path.join(BASE_DIR, 'cronJobChromeDriver.log')

executable_path = os.path.join(BASE_DIR, 'chromedriver_linux64/chromedriver')
extension_path = os.path.join(
    BASE_DIR, 'chromedriver_linux64/extension_2_8_9_0.crx')
print('executable_path', executable_path)
The Dan
  • 1,408
  • 6
  • 16
  • 41

4 Answers4

10

The bottom line is, No, doesn't supports extensions.

In the one of his comment, alexclarke@chromium.org mentioned:

I realize a lot of folks would like to use extensions with headless but unfortunately that's a large project which we have /no plans to do/. The problem is Headless Chromium is a content embedder which means it doesn't have access to anything from other content embedders such as chrome and unfortunately extensions are a chrome feature.

In another comment he further added, if you're using Selenium through DevTools you can build a proxy. Next you can filter URLs and modify headers via Network.setRequestInterception and Network.continueInterceptedRequest.

Reference

You can find a relevant detailed discussion in:

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352
2

this is now possible by modifying the following flag:

chrome_options.add_argument('--headless=chrome')

I tested it successfully.

I found it here: https://bugs.chromium.org/p/chromium/issues/detail?id=706008#c5

runbibot
  • 21
  • 1
1

Chrome does not support headless, but apparently Firefox does. Some relevant discussions:

  1. https://sqa.stackexchange.com/questions/32611/selenium-chromedriver-headless-chrome-failed-to-wait-for-extension-backgro
  2. Is it possible to run Google Chrome in headless mode with extensions?
0buz
  • 3,443
  • 2
  • 8
  • 29
0

If still having issue try using:

add_argument("--headless=new") instead of add_argument("--headless=chrome")

discussion is here: https://bugs.chromium.org/p/chromium/issues/detail?id=706008#c5

vovinio
  • 21
  • 2
  • Thank you for your answer. Could you please elaborate on what this code does and how it resolves the issue? Also, please include any important information from the link in the body of your answer. This is important in case the link does not work in the future. – Joshua Shew Aug 21 '23 at 09:46