Selenium Web Scraping (InvalidSelectorException)

Question

I am trying to web scrap a website with navigating the different pages.

I tried this code on another website that works for scrapping. I have now trying to scrap another website and failed.

It shows that "selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: An invalid or illegal selector was specified"

But I can't find the issue.. can anyone help to solve this.. Many thanks!!

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.utils import ChromeType

driver = webdriver.Chrome(
    service=Service(ChromeDriverManager(chrome_type=ChromeType.CHROMIUM).install())
)

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver.get("https://www.28hse.com/buy/residential/a1")

for i in range(3):
    all_cells = driver.find_elements(By.CSS_SELECTOR, ".item property_item")
    
    for cell in all_cells:
        names = cell.find_elements(By.CSS_SELECTOR, ".areaUnitPrice.separation").text

        if len(names) > 0:
            print(names[0].text)
        else:
            print("--- no name ---")

    next_btn = driver.find_element(By.CSS_SELECTOR, ".pagination_hi[2]")
    next_btn.click()
    try:
        print(f"Done page {i+1}")
        element = WebDriverWait(driver, 30).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, ".product-brief-wrapper"))
        )
    except:
        print("Wait failed")

What code line gives you the exception? – Prophet Feb 27 '22 at 20:35 — Prophet, Feb 27 '22 at 20:35

score 0 · Answer 1 · answered Feb 27 '22 at 20:46

0

This css_selector:

.pagination_hi[2]

definitely looks error prone. Possibly you intend to use:

.pagination

Additionally, you also need to change:

.item property_item

to

.item.property_item

answered Feb 27 '22 at 20:46

undetected Selenium

183,867
41
278
352

score 0 · Answer 2 · answered Feb 27 '22 at 20:59

As @undetected Selenium mentioned lots of error in your code.

1.

all_cells = driver.find_elements(By.CSS_SELECTOR, ".item property_item")

It should be when you are using css selector each class name should use .

 all_cells = driver.find_elements(By.CSS_SELECTOR, ".item.property_item")

names = cell.find_elements(By.CSS_SELECTOR, ".areaUnitPrice.separation").text

find_elements() returns list of elements and its not having property text

So this should be

names = cell.find_elements(By.CSS_SELECTOR, ".areaUnitPrice.separation")

pagination link

next_btn = driver.find_element(By.CSS_SELECTOR, ".pagination_hi[2]")

if you have two same elements present on the page and wants to select the second one using css selector you should use.

.pagination_hi:nth-of-type(2)

Ideally your code should like

next_btn = driver.find_element(By.CSS_SELECTOR, ".pagination_hi:nth-of-type(2)")

or you can use element uniquely identified on the page.

 next_btn = driver.find_element(By.CSS_SELECTOR, ".pagination_hi >a[rel='next']")

Thank you for the reply! What does that mean: (1) selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable Shall be related to the next_btn? (2) How am i suppose the scrap the text in .areaUnitPrice.separation? Much thanks!!!!!!!!!! — LVEEN, Feb 27 '22 at 21:20

Selenium Web Scraping (InvalidSelectorException)

2 Answers2