Trying to work with selenium and scrapy both

Question

I am to scrape dynamic website but the selenium will provide me these error 'chromedriver' executable needs to be in PATH Can you solve these problem

from scrapy import Spider
from scrapy.http import Request
from scrapy.utils.project import get_project_settings
from selenium import webdriver


class AuthorSpider(Spider):
    name = 'pushpa'


    def start_requests(self):
        self.driver = webdriver.Chrome(executable_path='C:/Program Files (x86)/chromedriver')
        driver = webdriver.Chrome(driver_path, options=options)
        driver.get('https://www.lazada.com.ph/shop-laptops/')
        link_elements = driver.find_elements_by_xpath(
            '//*[@data-qa-locator="product-item"]//a[text()]')

        for link in link_elements:
            yield{
                'url':link
            }

Duplicate of https://stackoverflow.com/questions/29858752/error-message-chromedriver-executable-needs-to-be-available-in-the-path — Eb J, Mar 03 '22 at 19:16

score 0 · Answer 1 · answered Mar 03 '22 at 15:51

0

executable_path should be set to absolute path to chromedriver.exe file containing the chromedriver.exe file itself.
So, in case your chromedriver.exe is inside the 'C:/Program Files (x86)/chromedriver' folder it should be

self.driver = webdriver.Chrome(executable_path='C:/Program Files (x86)/chromedriver/chromedriver.exe')

Also I don't understand why are you defining and initializing 2 objects of the driver? :

self.driver = webdriver.Chrome(executable_path='C:/Program Files (x86)/chromedriver')
driver = webdriver.Chrome(driver_path, options=options)

answered Mar 03 '22 at 15:51

Prophet

32,350
22
54
79

they will show me these error again – Amen Aziz Mar 03 '22 at 16:10
for what exactly code line? – Prophet Mar 03 '22 at 16:11
Traceback (most recent call last): File "e:\python39\lib\site-packages\scrapy\core\engine.py", line 129, in _next_request request = next(slot.start_requests) File "C:\Users\Dell\catalogue\catalogue\spiders\pushpa.py", line 12, in start_requests self.driver = webdriver.Chrome(executable_path='C:/Program Files (x86)/chromedriver/chromedriver.exe') File "e:\python39\lib\site-packages\selenium\webdriver\chrome\webdriver.py", line 73, in __init__ self.service.start() File "e:\python39\lib\site-packages\selenium\webdriver\common\service.py", line 81, in start – Amen Aziz Mar 03 '22 at 16:13
The error is still `executable needs to be in PATH`? Does the path to the file is `C:/Program Files (x86)/chromedriver/chromedriver.exe`? – Prophet Mar 03 '22 at 16:15
yes the error will same `executable needs to be in PATH` yes the path is in the C folder in the computer in program files – Amen Aziz Mar 03 '22 at 16:19
It is strange. Your IDE can not find the `chromedriver.exe` executable file in `'C:/Program Files (x86)/chromedriver/chromedriver.exe'` path. Maybe it is in `Program Files (x86)` folder itself or the folder it is in is not `/chromedriver/`? – Prophet Mar 03 '22 at 16:23

Trying to work with selenium and scrapy both

1 Answers1