I am trying to use selenium and headless chrome to scrape a website so since I don't have much experience I tried to test with this simple script but I seem to be getting the error below. Do you think there are any compatibility issues with the drivers?
Code:
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--window-size=1420,1080')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-gpu')
outputdir='~/zycus_report_scraper/data_engineering/scripts/python'
service_log_path = "{}/chromedriver.log".format(outputdir)
service_args = ['--verbose']
browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'/home/user/.local/bin/chromedriver', service_args=service_args, service_log_path=service_log_path)
browser.get('https://www.twitter.com')
print(browser.page_source.encode('utf-8'))
browser.quit()
Error:
Traceback (most recent call last):
File "zycus_selenium.py", line 12, in <module>
browser = webdriver.Chrome(chrome_options=chrome_options, executable_path=r'/home/groballe/.local/bin/chromedriver', service_args=service_args, service_log_path=service_log_path)
File "/home/groballe/.local/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
self.service.start()
File "/home/groballe/.local/lib/python3.6/site-packages/selenium/webdriver/common/service.py", line 98, in start
self.assert_process_still_running()
File "/home/groballe/.local/lib/python3.6/site-packages/selenium/webdriver/common/service.py", line 111, in assert_process_still_running
% (self.path, return_code)
selenium.common.exceptions.WebDriverException: Message: Service /home/user/.local/bin/chromedriver unexpectedly exited. Status code was: 1
Driver versions:
chromedriver version: ChromeDriver 80.0.3987.106 (f68069574609230cf9b635cd784cfb1bf81bb53a-refs/branch-heads/3987@{#882})
chromium-browser version: 80.0.3987.87
selenium: selenium-3.141.0
Edit: This is on a Windows Subsystem for Linux (WSL).