Getting Page Number from Site Using BeautifulSoup/Selenium

Question

I am learning web scraping and trying to get the total number of sale pages located at the bottom of this page ("https://www.farfetch.com/uk/shopping/women/sale/all/items.aspx"), but am struggling to do so.

To be precise, I am trying to get the text inside the element: <div class="_7ba1d5 _a535c4 _ec791b" data-testid="page-number">1 of 532</div>

I have tried using BeautifulSoup: Pages = pageSoup.find("div", {"data-testid" : "page-number"}).text but with no luck.

I then tried using Selenium but I am also struggling to find the class as well. I have tried using driver.find_element(By.XPATH('')) but with no luck as well.

Apologies if these are stupid questions but I am fairly new to web scraping.

score 0 · Answer 1 · answered Nov 16 '21 at 14:23

So if your page_source is correct you can get your goal with the following css selector:

soup.select_one('div[data-testid="page-number"]').text

Example

from bs4 import BeautifulSoup

html='''<div class="_7ba1d5 _a535c4 _ec791b" data-testid="page-number">1 of 532</div>'''
soup = BeautifulSoup(html, 'lxml')
soup.select_one('div[data-testid="page-number"]').text

score 0 · Accepted Answer · answered Nov 16 '21 at 14:45

Was able to extract the text through below code.

Need to scroll down to the end of the page so as to extract the details.

May or may not click on Accept Cookies button to proceed.

# Imports required for Explicit wait.

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver.get("https://www.farfetch.com/uk/shopping/women/sale/all/items.aspx")

wait = WebDriverWait(driver,30)

# Click on Accept cookies
wait.until(EC.element_to_be_clickable((By.XPATH,"//button[contains(text(),'Accept All')]"))).click()

# Scroll down to the footer
driver.execute_script("window.scrollBy(0,document.body.scrollHeight)")
driver.execute_script("window.scrollBy(0,-1000);")

page_num = wait.until(EC.presence_of_element_located((By.XPATH,"//div[@data-testid='page-number']")))
print(page_num.text)

1 of 532

score 0 · Answer 3 · answered Nov 16 '21 at 15:59

An ideal approach would be to keep scrolling untill your find the element with text as 1 of 532 inducing WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:

Using xpath:

driver.get("https://www.farfetch.com/uk/shopping/women/sale/all/items.aspx")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-testid='Button_PrivacySettingsBanner_AcceptAll']"))).click()
while True:
    try:
      driver.execute_script("window.scrollBy(0,1500)")
      print(WebDriverWait(driver, 5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div[data-testid='page-number']"))).text)
      break
    except TimeoutException:
      continue

driver.quit()

Console Output:
```
1 of 532
```

Getting Page Number from Site Using BeautifulSoup/Selenium

3 Answers3

Example