I'm trying to use selenium for web scraping, but I'm not able to load a site on the browser, this is my initial code:
import time
import os,sys
reload(sys)
sys.setdefaultencoding('utf8')
import re
from selenium import webdriver
from scrapy.http import TextResponse
url = 'https://www.nytimes.com/'
driver_1 = webdriver.Chrome()
driver_1.get(url)
With that code at get an error:
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH.
Which corresponds to this line:
driver_1 = webdriver.Chrome()
Hence I tried the whereis chromium-browser
command and got this paths:
/usr/bin/chromium-browser
/etc/chromium-browser
/usr/lib/chromium-browser
/usr/bin/X11/chromium-browser
/usr/share/chromium-browser
/usr/share/man/man1/chromium-browser.1.gz
But for each of them I get errors, for this one:
driver_1 = webdriver.Chrome("/usr/bin/chromium-browser")
I get the error:
selenium.common.exceptions.WebDriverException: Message: Service /usr/bin/chromium-browser unexpectedly exited. Status code was: 0
I'm using python 2.7X
EDIT:
I have found the answer at the correct path here:
driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")
For that to work it is required to run this command:
sudo apt-get install chromium-chromedriver