Unable to click HREF under headers (invisible elements)

Question

I am wanting to click all the Href tabs under the main headers and to navigate to those pages to scrape them. For speed of the job, I do am wanting to click the href without having to click the headers. My question is, is there a way to click these buttons even though it is not visible like the page on the right? It does not seem to be working for me. It seems to give me:

Traceback (most recent call last):
  File "C:/Users/Bain3/PycharmProjects/untitled4/Centrebet2.py", line 58, in <module>
    EC.element_to_be_clickable((By.XPATH, '(//*[@id="accordionMenu1_ulSports"]/li/ul/li/ul/li/a)[%s]' % str(index + 1)))).click()
  File "C:\Users\Bain3\Anaconda3\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

I have replaced

EC.element_to_be_clickable((By.XPATH, '(//*[@id="accordionMenu1_ulSports"]/li/ul/li/ul/li/a)[%s]' % str(index + 1)))).click()

with

driver.find_element_by_xpath('(//*[@id="accordionMenu1_ulSports"]/li/ul/li/ul/li/a)[%s]' % str(index + 1)).click()

This however does not seem to remedy it as it only clicks visible elements.

My code below is:

from random import shuffle
from selenium.webdriver.support.ui import WebDriverWait as wait

from selenium import webdriver as web
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import TimeoutException
from random import randint
from time import sleep
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
import csv
import requests
import time
from selenium import webdriver

success = False
while not success:
    try:
        driver = webdriver.Chrome()
        driver.set_window_size(1024, 600)
        driver.maximize_window()
        driver.get('http://centrebet.com/')
        success = True
    except:
        driver.quit()

sleep(5)

sports = driver.find_element_by_id("accordionMenu1_ulSports")
if sports.get_attribute("style") == "display: none;":
    driver.find_element_by_xpath('//ul[@id="menu_acc"]/li[3]/a').click()

driver.find_element_by_xpath(".//*[@data-type ='sports_l1'][contains(text(), 'Soccer')]").click()

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

options = driver.find_elements_by_xpath('//*[@id="accordionMenu1_ulSports"]/li/ul/li/ul/li/a')

# Get list of inetegers [1, 2, ... n]
indexes = [index for index in range(len(options))]
# Shuffle them
shuffle(indexes)
for index in indexes:
    # Click on random option
    wait(driver, 10).until(
        EC.element_to_be_clickable((By.XPATH, '(//*[@id="accordionMenu1_ulSports"]/li/ul/li/ul/li/a)[%s]' % str(index + 1)))).click()

I have also tried:

driver.execute_script('document.getElementByxpath("//*[@id="accordionMenu1_ulSports"]/li/ul/li/ul/li/a").style.visibility = "visible";') 

To remedy this. Though this simply gives an error. Any ideas on how to resolve this issue of invisible elements?

Answer 1

You can even try using JavascriptExecutor.

Use below code to make your style attribute = display:block;

driver.execute_script("arguments[0].style.display = 'none'", driver.find_element_by_xpath("//*[@id='accordionMenu1_ulSports']/li/ul/li/ul"))

Note : Make sure you are using correct xpath. your <ul> element is hidden not <a> so so take xpath of that <ul> tag only and try

Answer 2

driver.execute_script('document.getElementByxpath("//*[@id="accordionMenu1_ulSports"]/li/ul/li/ul/li/a").style.visibility = "visible";')

gives you error because it's not correct way to use XPath in Javascript. Correct way you can find here

To scrape required data you can use below code:

import requests
import time
from selenium import webdriver

url = "http://centrebet.com/"
success = False
while not success:
try:
    driver = webdriver.Chrome()
    driver.set_window_size(1024, 600)
    driver.maximize_window()
    driver.get(url)
    success = True
except:
    driver.quit()

time.sleep(5)

sports = driver.find_element_by_id("accordionMenu1_ulSports")
links = [url + link.get_attribute("onclick").replace("menulink('", "").replace("')", "") for link in sports.find_elements_by_xpath('.//a[starts-with(@onclick, "menulink")]')]
for link in links:
    print(requests.get(link).text)

Instead of clicking on each link, you can request content of each page with HTTP-GET