Requests not working as intended and Selenium not installed properly

Question

I am trying to import the html code for a certain website (https://www.trendyol.com/de/s/alle-marken). Since requests didn't work as intended (not finding all the code), I was advised to try to download Selenium to try a different approach. My first question is, why did requests not find all the code? I was told it has something to do with JavaScript, but I am not familiar with the language and didn't understand the explanation. My second problem is that I am having problems installing selenium and the chrome web driver. I used python3.11 -m pip install selenium and python3.11 -m pip install webdriver-manager to install each package, and downloaded matching chrome and chromedriver versions (116.0.5829.0 (r1156474)) from this website: https://googlechromelabs.github.io/chrome-for-testing/

I also put all the files from the new chrome installation as well as the webdriver folder in this path: C:\Program Files\Google\Chrome\Application

However, when I try to run the code given in the website:

import time
from selenium import webdriver

driver = webdriver.Chrome('C:\Program Files\Google\Chrome\Application\chromedriver-win64\chromedriver.exe')
driver.get('http://www.google.com/')

time.sleep(5)  # Let the user actually see something!

search_box = driver.find_element_by_name('q')
search_box.send_keys('ChromeDriver')
search_box.submit()

time.sleep(5)  # Let the user actually see something!

driver.quit()

I get the following error message:

Traceback (most recent call last):
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\selenium\webdriver\common\driver_finder.py", line 42, in get_path
    path = SeleniumManager().driver_location(options) if path is None else path
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\selenium\webdriver\common\selenium_manager.py", line 74, in driver_location
    browser = options.capabilities["browserName"]
              ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'capabilities'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\Users\Guilherme Candeias\Downloads\scraper.py", line 4, in <module>
    driver = webdriver.Chrome('C:\Program Files\Google\Chrome\Application\chromedriver-win64\chromedriver.exe')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\selenium\webdriver\chrome\webdriver.py", line 47, in __init__
    self.service.path = DriverFinder.get_path(self.service, self.options)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\selenium\webdriver\common\driver_finder.py", line 44, in get_path
    raise NoSuchDriverException(f"Unable to obtain {service.path} using Selenium Manager; {err}")
selenium.common.exceptions.NoSuchDriverException: Message: Unable to obtain chromedriver using Selenium Manager; 'str' object has no attribute 'capabilities'; For documentation on this error, please visit: https://www.selenium.dev/documentation/webdriver/troubleshooting/errors/driver_location

I tried installing other versions of chrome, but could not find chromedriver versions that support my system (win64). I also tried using:

from webdriver_manager.chrome import ChromeDriverManager

driver = webdriver.Chrome(ChromeDriverManager().install())

But I only got a different error message:

Traceback (most recent call last):
  File "c:\Users\Guilherme Candeias\Downloads\trendyol scraper.py", line 5, in <module>
    driver = webdriver.Chrome(ChromeDriverManager().install())
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\webdriver_manager\chrome.py", line 39, in install
    driver_path = self._get_driver_path(self.driver)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\webdriver_manager\core\manager.py", line 30, in _get_driver_path
    file = self._download_manager.download_file(driver.get_driver_download_url())
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\webdriver_manager\drivers\chrome.py", line 40, in get_driver_download_url
    driver_version_to_download = self.get_driver_version_to_download()
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\webdriver_manager\core\driver.py", line 51, in get_driver_version_to_download
    self._driver_to_download_version = self._version if self._version not in (None, "latest") else self.get_latest_release_version()
                                                                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\webdriver_manager\drivers\chrome.py", line 62, in get_latest_release_version
    resp = self._http_client.get(url=latest_release_url)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\webdriver_manager\core\http.py", line 37, in get
    self.validate_response(resp)
  File "C:\Users\Guilherme Candeias\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\webdriver_manager\core\http.py", line 16, in validate_response
    raise ValueError(f"There is no such driver by url {resp.url}")
ValueError: There is no such driver by url https://chromedriver.storage.googleapis.com/LATEST_RELEASE_116.0.5829

dont ask multiple questions in one thread. And how do you know requests dont work when you didnt install selenium properly? — Rolandas Ulevicius, Jun 19 '23 at 05:19
Oh sorry, my first question was more of a curiosity so I didn't want to open a new thread. The reason I think requests isn't getting the full html is because when I use beautifulsoup to use find_all("a"), it only finds a couple elements at the start of the page. My internet is working fine because the status code for this website is 200. — Guilherme Candeias, Jun 19 '23 at 07:19

score 0 · Accepted Answer · answered Jun 19 '23 at 08:42

If you selenium version is v4.6.0 or above, you don't really need to set the path of chromedriver.exe anymore.

So your code can be reduced to below:

driver = webdriver.Chrome()
driver.get('http://www.google.com/')

Refer this - https://stackoverflow.com/a/76463081/7598774

Requests not working as intended and Selenium not installed properly

1 Answers1