Click on Load more element using Selenium and Python

Question

I have a news website to scrape. But after scraping the first page, I need to click on "load more" which is a link to view other news and then scrape. Where I am encountering an issue is that I can't seem to click on "load more" because it doesn't have a "href". I have tried all possible resources, but they have all worked to no avail. Please, kindly help me out with this. Thank you. I have attached the code I used and the link to the website.

Code trials:

pip install webdriver-manager

from selenium.webdriver.common.keys import Keys
import time
from webdriver_manager.chrome import ChromeDriverManager
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException

driver = webdriver.Chrome(ChromeDriverManager().install())
url = 'https://www.legit.ng/tag/road-accidents-in-nigeria/'
driver.get(url)
link = driver.find_element_by_link_text("Entertainment")
link.click()

Answer 1

To click on the element with text as Load more you need to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

• Using XPATH and text()[contains()]:

  driver.get("https://www.legit.ng/tag/road-accidents-in-nigeria/")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[text()[contains(.,'Load more')]]"))).click()
  

• Using XPATH and contains():

  driver.get("https://www.legit.ng/tag/road-accidents-in-nigeria/")
  WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[contains(., 'Load more')][./p[text()]]"))).click()
  

• Note: You have to add the following imports :

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC