Clicking through multiple links using selenium

Question

Trying to break a bigger problem I have into smaller chunks main question

I am currently inputting a boxer's name into an autocomplete box, selecting the first option that comes up (boxer's name) then clicking view more until I get a list of all the boxer's fights and the view more button stops appearing.

I am then trying to create a list of onclick hrefs I would like to click then iteratively clicking on each and getting the html from each page/fight. I would ideally want to extract the text in particular.

This is the code I have written:

page_link = 'http://beta.compuboxdata.com/fighter'
chromedriver = 'C:\\Users\\User\\Downloads\\chromedriver'
cdriver = webdriver.Chrome(chromedriver)
cdriver.maximize_window()
cdriver.get(page_link)
wait = WebDriverWait(cdriver,20)
wait.until(EC.visibility_of_element_located((By.ID,'s2id_autogen1'))).send_keys('Deontay Wilder')
wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'select2-result-label'))).click()
while True:
    try:
        element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'view_more'))).click()
    except TimeoutException:
        break
# fighters =  cdriver.find_elements_by_xpath("//div[@class='row row-bottom-margin-5']/div[2]")
links = [x.get_attribute('onclick') for x in wait.until(EC.visibility_of_element_located((By.XPATH, "//*[contains(@onclick, 'get_fight_report')]/a")))]
htmls = []
for link in links:
    cdriver.get(link)
    htmls.append(cddriver.page_source)

Running this however gives me the error message:

ElementClickInterceptedException          Traceback (most recent call last)
<ipython-input-229-1ee2547c0362> in <module>
     10 while True:
     11     try:
---> 12         element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'view_more'))).click()
     13     except TimeoutException:
     14         break
ElementClickInterceptedException: Message: element click intercepted: Element <a class="view_more" href="javascript:void(0);" onclick="_search('0')"></a> is not clickable at point (274, 774). Other element would receive the click: <div class="col-xs-12 col-sm-12 col-md-12 col-lg-12">...</div>
  (Session info: chrome=78.0.3904.108)

UPDATE

I have tried looking at a few answers with similar error messages and tried this

while True:
    try:
        element = cdriver.find_element_by_class_name('view_more')
        webdriver.ActionChains(cdriver).move_to_element(element).click(element).perform()
#         element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'view_more'))).click()
    except TimeoutException:
        break
links = [x.get_attribute('onclick') for x in wait.until(EC.visibility_of_element_located((By.XPATH, "//*[contains(@onclick, 'get_fight_report')]/a")))]
htmls = []
for link in links:
    cdriver.get(link)
    htmls.append(cddriver.page_source)

but this seems to create some sort of infinite loop at the ActionChains point. Seems to be constantly waiting for the view more href to appear

[This question](https://stackoverflow.com/questions/48665001/can-not-click-on-a-element-elementclickinterceptedexception-in-splinter-selen) has a similar error message to yours. What happens if you try to implement the top voted answer? — Kevin, Dec 05 '19 at 15:27
@Kevin tried it, appears to create what resembles an infinite loop. Explained it in my question — Emm, Dec 05 '19 at 16:03
I notice that the element has an attribute `onclick="_search('0')`. Maybe you could skip pressing the button entirely, and just execute `_search('0')` directly? I'm not sure if Selenium can do that, but it's worth looking into. — Kevin, Dec 05 '19 at 17:45

score 0 · Answer 1 · answered Dec 05 '19 at 16:18

0

click function should already move the window so the element is in the viewable window. So you don't need that action chain (I think...) but the original error shows some other element OVER the view more button.

You may need to remove (or hide) this element from the DOM, or if it's a html window, "dismiss" it. So pinpointing this covering element is key and then deciding on a strategy to uncover the view more button.

Your site http://beta.compuboxdata.com/fighter doesn't seem to be working at the time so I can't dig in further.

answered Dec 05 '19 at 16:18

DMart

2,401
1
14
19

I see, not sure how to do this, site seems to be working now – Emm Dec 05 '19 at 16:20
ok, there's definitely a banner ad that may be getting in the way of the link. Easiest way is to identify and delete it from the dom – DMart Dec 05 '19 at 16:35
but doesn't visibility_of_element wait for the element to be available in DOM before I click it. I am referring to clicking through the view more hrefs (which the code does)..maybe i'm misreading your reply? – Emm Dec 05 '19 at 16:48
"Visibility means that the element is not only displayed but also has a height and width that is greater than 0." https://selenium-python.readthedocs.io/api.html#selenium.webdriver.support.expected_conditions.visibility_of_element_located – DMart Dec 05 '19 at 17:57

Clicking through multiple links using selenium

1 Answers1