Selenium,unable to get price from website while running on google cloud platform windows virtual machine

Question

The code:

from bs4 import BeautifulSoup
import undetected_chromedriver as uc
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = uc.ChromeOptions()
options.add_argument('--blink-settings=imagesEnabled=false') # disable images for loading of page faster
options.add_argument('--disable-notifications')
prefs = {"profile.default_content_setting_values.notifications" : 2}
options.add_experimental_option("prefs",prefs)
driver = uc.Chrome(options=options)

driver.get('https://www.hepsiburada.com/pinar-tam-yagli-sut-4x1-lt-pm-zypinar153100004')

try:
 price_element = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.ID, 'offering-price')))
        price = float(price_element.find_element_by_xpath('./span[1]').text + '.' + price_element.find_element_by_xpath('./span[2]').text)
      
except:
  print("there is no price info...")

Hello friends, what could be the reason why I can't get the price information while working on the gcp virtual machine, how can I fix the problem in data extraction? While working on a normal computer, I was getting the price information with the same code. What changes when you work in a virtual machine?

On virtual computer it throws AttributeError: 'WebElement' object has no attribute 'find_element_by_xpath'. A normal computer doesn't either. Both have chrome installed. I also turned off the firewall on the virtual computer.

I also added these to my code but nothing changed:

options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-gpu')
options.add_argument('--disable-extensions')
options.add_argument('--disable-notifications')
options.add_argument('--disable-popup-blocking')

Version Informations:

OS Name Microsoft Windows Server 2022 Datacenter

Google Chrome Version 111.0.5563.65 (Official Build) (64-bit)

Jupyterlab 3.4.4

score 3 · Accepted Answer · answered Mar 23 '23 at 07:09

I suspect the problem is that you have different versions of selenium installed on your virtual machine and personal machine. You can confirm this by running this script on both machines and comparing the results:

import selenium
print(selenium.__version__)

As this great answer explains, selenium has removed the find_element_by_* methods in versions 4.3.0 and above. Rather than downgrading the selenium installation on the virtual machine, you should upgrade the selenium installation on your personal machine to match the version on the virtual machine (python3 -m pip install --force-reinstall "selenium==[version]"), and update your code to use the find_element method. Your updated code should look something like this:

price = float(price_element.find_element('xpath', './span[1]').text + '.' + price_element.find_element('xpath', './span[2]').text)

correct! One more thing worth mentioning is that client and server side (webdriver) versions need to be compatible, so one may need to upgrade Webdriver (serverside) too after upgrading a Webdriver client — Antonio, Mar 28 '23 at 19:58

undetected Selenium · Answer 2 · 2023-03-23T23:03:27.753

This error message...

AttributeError: 'WebElement' object has no attribute 'find_element_by_xpath'

...implies that using selenium4 find_element_by_* commands are deprecated. You have to use find_element() instead.

Solution

To get the price from the website ideally you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:

Code Block:

driver.get("https://www.hepsiburada.com/pinar-tam-yagli-sut-4x1-lt-pm-zypinar153100004")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button#onetrust-accept-btn-handler"))).click()
try:
  price_element = WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'span#offering-price')))
  price = float(price_element.find_element(By.CSS_SELECTOR, "span[data-bind*='currentPriceBeforePoint']").text + '.' + price_element.find_element(By.CSS_SELECTOR, "span[data-bind*='currentPriceAfterPoint']").text)
  print(price)
except:
  print("there is no price info...")
driver.quit()

Console Output:
```
157.95
```

Selenium,unable to get price from website while running on google cloud platform windows virtual machine

2 Answers2

Solution