1

I am web-scraping reviews from Goodreads for a project. Here's an example of a page I've been trying: https://www.goodreads.com/book/show/2767052-the-hunger-games/reviews?

The reviews page initially shows 30 reviews with a 'Show More' button at the bottom. Selenium seems unable to click the button.

Here is the code I'm using:

showmore_button = driver.find_element(By.XPATH, '/html/body/div[1]/div/main/div[1]/div[2]/div[4]/div[4]/div/button/span[1]')
driver.execute_script("arguments[0].click();", showmore_button)

I have also tried showmore_button.click() but that leads to an exception stating that the element is not clickable

For more context my driver is set up like this:

def createdriver():
    options = webdriver.ChromeOptions()

    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--disable-dev-shm-usage')
    options.add_argument("start-maximized")
    options.add_argument('--window-size=1920,1080')  
    options.add_argument("--incognito")
    
    driver = webdriver.Chrome(options=options)

    return driver

and then I use:

driver = createdriver()
driver.get(url)

Where the URL is the reviews page I'm trying to scrape

Alex
  • 37
  • 6
  • Where did you get this path from and why are you clicking not on the button, but on the span element? – Сергей Кох Feb 05 '23 at 19:43
  • @СергейКох I have also tried the button but unfortunately it led to the same result. I inspected the page using Chrome's built in tools and copied the full xpath from the element – Alex Feb 05 '23 at 19:56
  • I got this path - `/html/body/div[1]/div/main/div[1]/div[2]/div[5]/div[4]/div/button`. It might be better to find the button by `class="Button Button--secondary Button--small"` – Сергей Кох Feb 05 '23 at 20:11
  • @Alex please share your code make this request request – Ahmed Yousif Feb 05 '23 at 20:11
  • @СергейКох I tried both but it was unable to locate element for either option – Alex Feb 05 '23 at 20:16

2 Answers2

0

As the page loads for few milli-seconds before the element is displayed, you need to apply selenium waits. Try using implicit wait after creating the driver instance, see code below:

driver = createdriver()
driver.implicitly_wait(10)
driver.get(url)

Above code waits for 10 seconds searching for the element before throwing error

Also another suggestion: Instead of using an absolute XPath, as a best practice use relative XPath. This is because relative XPath is more consistent compared to absolute XPath. Absolute XPath may stop working, If the DOM structure changes in the future. Try the below relative XPath:

showmore_button = driver.find_element(By.XPATH, '//span[contains(text(),"Show more reviews")]')
driver.execute_script("arguments[0].click();", showmore_button)
Shawn
  • 4,064
  • 2
  • 11
  • 23
0

To click on the element Show more reviews at the bottom of the page you need to scrollIntoView() inducing WebDriverWait for the visibility_of_element_located() and you can use the following locator strategies:

  • Code block:

    driver.get('https://www.goodreads.com/book/show/2767052-the-hunger-games/reviews?')
time.sleep(5)
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='ReviewsList__listContext ReviewsList__listContext--centered']//span[contains(., 'Displaying 1 -')]"))))
driver.execute_script("arguments[0].click();", driver.find_element(By.XPATH, "//span[text()='Show more reviews']"))

  • Note: You have to add the following imports :

    from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

  • Browser snapshot:

Show more reviews

undetected Selenium
  • 183,867
  • 41
  • 278
  • 352