Selenium not able to scrape content by css selector and xpath

Question

I try to scrape product titles from https://www.ternbicycles.com/us/bikes#

I try to use xpath and css selector, but the find_elements() function only returns empty list. I didn't find framesets from this page and use execute_script to scrolled the window down to bottom... oh and I did set up wait time

But still no use....

Could some one help me out. Maybe could help me to try this url and give me some idea Thank you

URL='https://www.ternbicycles.com/us/bikes'
driver.implicitly_wait(10)
driver.get(URL)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
print(driver.title)
titles=driver.find_elements(By.CSS_SELECTOR,"//a[normalize-space()='Quick Haul P9 Performance']")

print(titles)

#result: Just a moment... []

############################################################# Update at 14:05 08/26/2022 Thank you guys for your reply,,,

i am using colab and i got TimeoutException: Message: Stacktrace... think should be a configuration problem? Do you have any solution?

Pulling web elements using Selenium on Colab. error "TimeoutException: Message: " (Working fine in VSC)

Here is what i imported before i ran your codes

!apt-get update
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
!pip install selenium
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
#set up Chrome driver
options=webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
options.add_argument("window-size=1280,720")
#Define web driver as a Chrome driver
driver=webdriver.Chrome('chromedriver',options=options)
driver.implicitly_wait(10)

score 1 · Answer 1 · answered Aug 26 '22 at 19:54

1

The webpage isn't dynamic. So you can parse all the required contents/data with the help of bs4 and requests module.

Example:

import requests
from bs4 import BeautifulSoup
import pandas as pd
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:94.0) Gecko/20100101 Firefox/94.0'}
r = requests.get('https://www.ternbicycles.com/us/bikes',headers =headers)
print(r)
soup = BeautifulSoup(r.text,'html.parser')
for card in soup.select('div[class="grid grid-cols-1 md-grid-cols-2 lg-grid-cols-3 gap-4 md:grid-cols-2 lg:grid-cols-3"] > div'):
    title = card.select_one('h2[class="font-header-bold pb-4"] > a').get_text(strip=True)
    print(title)

Output:

Quick Haul P9 Performance
Quick Haul D8
NBD S5i
NBD P8i
Short Haul D8
Vektron S10
HSD S11
HSD S8i
HSD P9
Verge X11
Verge S8i
Verge D9
Node D8
Node D7i
BYB P8
Vektron Q9
HSD S+
HSD P9 Performance
GSD R14
GSD S00 LX
GSD S10 LX
GSD S10
Verge P10
Eclipse X22
Eclipse P20
Eclipse D16
Link D7i
Link D8
Link C8
Link A7
BYB S11

answered Aug 26 '22 at 19:54

Md. Fazlul Hoque

15,806
5
12
32

I try to use your code and get and i even try to switch free proxy in the request.get(url,headers,proxy) but still the same.... is there a way=(( – kaispace30098 Aug 26 '22 at 20:14
No need to use proxy. Just use your real-user agent as header like me. That's enough. – Md. Fazlul Hoque Aug 26 '22 at 20:24
There is a tag called web-scraping – Md. Fazlul Hoque Aug 26 '22 at 20:27
I got Example response — The HTTP 403 Forbidden response status code indicates that the server understands the request but refuses to authorize it. – kaispace30098 Aug 26 '22 at 21:38
is it the website bans my ip?? – kaispace30098 Aug 26 '22 at 21:38
Could be that, @kaispace30098. If you will try running the code locally from your computer, it should work. – Barry the Platipus Aug 26 '22 at 21:39
i am trying local machine, and downloaded the chromer driver @platipus_on_fire, but is the path where i save, is the chromedriver.exe or the chromedriver folder thank you – kaispace30098 Aug 26 '22 at 22:21
webdriver_service = Service('path for chromedriver.exe or folder') – kaispace30098 Aug 26 '22 at 22:21
i think i got it Service(executable_path, port, service_args, …) – kaispace30098 Aug 26 '22 at 22:29
@platipus_on_fire But it shows: SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 101 Current browser version is 104.0.5112.102 – kaispace30098 Aug 26 '22 at 22:30
change the browser to 104 and the problem is solved now! – kaispace30098 Aug 26 '22 at 22:48

score 1 · Accepted Answer · answered Aug 26 '22 at 20:22

The following selenium based solution works (tested):

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument('disable-notifications')
chrome_options.add_argument("window-size=1280,720")

webdriver_service = Service("chromedriver/chromedriver") ## path to where you saved chromedriver binary
browser = webdriver.Chrome(service=webdriver_service, options=chrome_options)

url = 'https://www.ternbicycles.com/us/bikes' 

browser.get(url) 

try:
    WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".agree-button.eu-cookie-compliance-agree-button"))).click()
    print('accepted cookies')
except Exception as e:
    print('no cookies for you')
browser.execute_script("window.scrollTo(0,document.body.scrollHeight);")
bikes = WebDriverWait(browser, 20).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, "h2[class='font-header-bold pb-4']")))
for b in bikes:
    print(b.text)

This will print out:

accepted cookies
Quick Haul P9 Performance
Quick Haul D8
NBD S5i
NBD P8i
Short Haul D8
Vektron S10
HSD S11
HSD S8i
HSD P9
Verge X11
Verge S8i
Verge D9
Node D8
Node D7i
BYB P8
Vektron Q9
HSD S+
HSD P9 Performance
GSD R14
GSD S00 LX
GSD S10 LX
GSD S10
Verge P10
Eclipse X22
Eclipse P20
Eclipse D16
[...]

Selenium setup is for python, you can adapt the code to your own setup, just note the imports and the code after defining the browser (driver). Selenium docs: https://www.selenium.dev/documentation/

thank you for your help! I think it is because i am using colab....i put the what i install and import in colab could you help me to get rid off the error thank you TimeoutException: Message: — kaispace30098, Aug 26 '22 at 21:12
If you are using G Colab, you should use headless option for chromedriver @kaispace30098. Or better, use F.Hoque solution. Also, make sure you update install ed packages with pip (also update pip), and restart the kernel. — Barry the Platipus, Aug 26 '22 at 21:17
Yes i did use headless for option the update above have all what i installed and imported — kaispace30098, Aug 26 '22 at 21:33

score 0 · Answer 3 · answered Aug 26 '22 at 20:21

0

To extract the name of the product titles you have to induce WebDriverWait for visibility_of_all_elements_located() and using List Comprehension you can use either of the following locator strategies:

Using CSS_SELECTOR and text attribute:

driver.get('https://www.ternbicycles.com/us/bikes#')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.agree-button.eu-cookie-compliance-agree-button"))).click()
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.title>h2>a")))])

Using XPATH and get_attribute("innerHTML"):

driver.get('https://www.ternbicycles.com/us/bikes#')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[@class='agree-button eu-cookie-compliance-agree-button']"))).click()
print([my_elem.get_attribute("innerHTML") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='title']/h2/a")))])

Console Output:

['Quick Haul P9 Performance', 'Quick Haul D8', 'NBD S5i', 'NBD P8i', 'Short Haul D8', 'Vektron S10', 'HSD S11', 'HSD S8i', 'HSD P9', 'Verge X11', 'Verge S8i', 'Verge D9', 'Node D8', 'Node D7i', 'BYB P8', 'Vektron Q9', 'HSD S+', 'HSD P9 Performance', 'GSD R14', 'GSD S00 LX', 'GSD S10 LX', 'GSD S10', 'Verge P10', 'Eclipse X22', 'Eclipse P20', 'Eclipse D16', 'Link D7i', 'Link D8', 'Link C8', 'Link A7', 'BYB S11']

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

answered Aug 26 '22 at 20:21

undetected Selenium

183,867
41
278
352

Thank you for the help, i didn't click the cookie request button, good to know, but still not working, i am using colab to run python, and i posted all my configuration above, if you are willing to help, could you help me to check – kaispace30098 Aug 26 '22 at 21:07
_but still not working_: Do you see any error? Possibly you are executing your test as a root user, can you execute as a non-root user? – undetected Selenium Aug 26 '22 at 21:12
https://colab.research.google.com/drive/1m5lQzfVA8QFCtjTYQOTuUXVxDRzFpLWi?usp=sharing Could you open the link? i got TimeoutException: Message: Stacktrace – kaispace30098 Aug 26 '22 at 21:17
Yup, I can open. But I don't see any error. How can I execute? – undetected Selenium Aug 26 '22 at 21:21
Sorry i grant you edit – kaispace30098 Aug 26 '22 at 21:23
just edit whatever may work for you thank you!!! the second cell is your code, and the first cell is the library i installed, there is no problem to scrape other sites, but i dont know why i cannot do this site so i am asking you guys.. – kaispace30098 Aug 26 '22 at 21:23
Just change to `driver=webdriver.Chrome(executable_path='chromedriver', options=options)` copy at https://colab.research.google.com/drive/1PCijORPlZWJZ-wH0bNgaWNdpA5VBBXLD – undetected Selenium Aug 26 '22 at 21:26
no.....still have the error https://colab.research.google.com/drive/1m5lQzfVA8QFCtjTYQOTuUXVxDRzFpLWi#scrollTo=8bY8Q5ZA9CZn – kaispace30098 Aug 26 '22 at 21:31
Change as `driver=webdriver.Chrome(executable_path='/usr/bin/chromedriver', options=options)` or `driver=webdriver.Chrome(executable_path='/usr/lib/chromium-browser/chromedriver', options=options)` – undetected Selenium Aug 26 '22 at 21:39
I tried both options still have timeout exception error https://colab.research.google.com/drive/1m5lQzfVA8QFCtjTYQOTuUXVxDRzFpLWi#scrollTo=8bY8Q5ZA9CZn – kaispace30098 Aug 26 '22 at 21:44

Selenium not able to scrape content by css selector and xpath

3 Answers3