Q: Can't keep clicking button by chorme web driver in python

Question

Question:

How can I show all reviews by clicking "Show more reviews" button?

What did I do:

1. To scrape all reviews, I decided to Keep clicking until that button disappears.
2. But some new reviews didn't appear after clicking for 8 times (using while below).
3. I already checked xpath of the button but it didn't changed.
4. If I click the button manually without the driver, I can see new reviews.

import time, random
from selenium import webdriver
from latest_user_agents import get_random_user_agent

### ser user_agent
user_agent = get_random_user_agent()
options = webdriver.ChromeOptions()
options.add_argument("--user-agent=" + user_agent)
driverpath = "C:/Users/~~/chromedriver.exe"
driver = webdriver.Chrome(chrome_options=options,executable_path=driverpath)

### target url
url = "https://www.sony.co.uk/electronics/truly-wireless/wf-l900/reviews-ratings"

### open URL
driver.get(url)
time.sleep(5)

# accept cookies
driver.find_element_by_xpath('//*[@id="onetrust-accept-btn-handler"]').click()

# show all reviews
try:
    while True:
        driver.execute_script('window.scroll(0,1000000);') 
        driver.find_element_by_xpath('//*[@id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button').click()
        time.sleep(random.randrange(2, 5, 1))
except:
    print("---------------- finish showing all reviews ----------------")

Answer 1

Try going to the end of the webpage and then do it as the button might not been loaded fully.

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

You might also try to press the button only when it is available.

element = WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.ID, "myElement")))

Answer 2

You can try javascript click:

element = driver.find_element_by_xpath('//*[@id="reviews_listing_278405943492055451029662"]/div[3]/div[4]/button')
driver.execute_script("arguments[0].click();", element)

In this method, the scrolling shouldn't be necessary. See more info about the subject here.