0

I have a list with a few boxers. I have created a function that will use selenium to open a particular web page, and then type in the name of a given boxer in my list on an autocomplete box. I then click on the first name that appears and then click on a view more (href). I continue clicking on view more until I get to the bottom of the page

I then read all the text with list of fights the boxer has fought and append to a dataframe column. I have tested this code with one boxer and know that it definitely works up until this point.

I then click on a download stats button for each fight in the list of fights I have collected and get text from the form that pops up. I then append all this information to another column.

This is the code I have written:

boxer_list = ['Deontay Wilder','Tyson Fury','Andy Ruiz']
def get_boxers(boxers):
    page_link = 'http://beta.compuboxdata.com/fighter'
    chromedriver = 'C:\\Users\\User\\Downloads\\chromedriver'
    cdriver = webdriver.Chrome(chromedriver)
    cdriver.maximize_window()
    cdriver.get(page_link)
    wait = WebDriverWait(cdriver,10)
    wait.until(EC.visibility_of_element_located((By.ID,'s2id_autogen1'))).send_keys(str(boxers))
    wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'select2-result-label'))).click()
    while True:
        try:
            element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'view_more')))
            element.click()
        except TimeoutException:
            break
    fighters =  cdriver.find_elements_by_xpath("//div[@class='row row-bottom-margin-5']/div[2]")
    cols = ['fighters', 'stats']
    fight_data = pd.DataFrame(columns = cols)
    for fighter in fighters:
        fight_data.append({'fighters':fighter.text},ignore_index=True)
        cdriver.find_element_by_xpath("//a[contains(@onclick,'get_fight_report')]").click()
        punches = cdriver.find_elements_by_xpath("//div[@class='modal-content']")
        for punch in punches:
            fight_data.append({'stats':punch.text},ignore_index=True)
    cdriver.refresh()
    return fight_data
boxing_dict = {boxer.replace(' ',''):pd.DataFrame([get_boxers(boxer)]) for boxer in boxer_list}

However when I run my function I get the error message:

ElementClickInterceptedException          Traceback (most recent call last)
<ipython-input-204-1e27dbb96927> in <module>
     35     cdriver.refresh()
     36     return fight_data
---> 37 boxing_dict = {boxer.replace(' ',''):pd.DataFrame([get_boxers(boxer)]) for boxer in boxer_list}
..............
ElementClickInterceptedException: Message: element click intercepted: Element <a class="view_more" href="javascript:void(0);" onclick="_search('0')">...</a> is not clickable at point (274, 774). Other element would receive the click: <div class="col-xs-2 col-sm-2 col-md-2 col-lg-2">...</div>
  (Session info: chrome=78.0.3904.108)

So from what I am gathering the code works up until this point

    for fighter in fighters:
        fight_data.append({'fighters':fighter.text},ignore_index=True)

I cannot, while looping through each of the fights in the list, click on the download button and get the punch stats from the pop up. This appears to be were the error message is coming from (I assume)

UPDATE:

I have tried updating my for loop to to wait before clicking on the view download button for each fight in my loop

for fighter in fighters:
    fight_data.append({'fighters':fighter.text},ignore_index=True)
    elem = wait.until(EC.element_to_be_clickable((By.XPATH,"//a[contains(@onclick,'get_fight_report')]")))
    elem.click()
    punches = cdriver.find_elements_by_xpath("//div[@class='modal-content']")
    for punch in punches:
        fight_data.append({'stats':punch.text},ignore_index=True)

Running this with a hardcoded key in send_keys() still returns an empty dataframe

Emm
  • 2,367
  • 3
  • 24
  • 50

1 Answers1

0
wait.until(EC.visibility_of_element_located((By.ID,'s2id_autogen1'))).send_keys('Deontay Wilder')

Is it intentional 'Deontay Wilder' hardcoded in your function definition?

Edit

till now I can only say you to change you code with this line (added another row in class, without it fighters returned empty list

    fighters =  cdriver.find_elements_by_xpath("//div[@class='row row-bottom-margin-5']/div[2]")

I can print fighter.text values in console, but I can't append nothing to fight_data, neither an hardcoded string. I don't understand why...

Edit2

boxer_list = ['Deontay Wilder','Tyson Fury','Andy Ruiz']
def get_boxers(boxers):
    page_link = 'http://beta.compuboxdata.com/fighter'
    chromedriver = 'C:\\Users\\User\\Downloads\\chromedriver'
    cdriver = webdriver.Chrome(chromedriver)
    cdriver.maximize_window()
    cdriver.get(page_link)
    wait = WebDriverWait(cdriver,10)
    wait.until(EC.visibility_of_element_located((By.ID,'s2id_autogen1'))).send_keys(str(boxers))
    wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'select2-result-label'))).click()
    while True:
        try:
            element = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'view_more')))
            element.click()
        except :
            break
    fighters =  cdriver.find_elements_by_xpath("//div[@class='row row-bottom-margin-5']/div[2]")
    cols = ['fighters', 'stats']
    fight_data = pd.DataFrame(columns = cols)
    for fighter in fighters:
        fight_data=fight_data.append({'fighters':fighter.text},ignore_index=True)
        cdriver.find_element_by_xpath("//a[contains(@onclick,'get_fight_report')]").click()
        punches = cdriver.find_elements_by_xpath("//div[@class='modal-content']")
        for punch in punches:
            fight_data=fight_data.append({'stats':punch.text},ignore_index=True)
    cdriver.refresh()
    return fight_data

I don't know how you want to collect stats. it's pretty a giant form and so punch.text return empty list. The first part in the meanwhile is solved

Enrico Belvisi
  • 94
  • 7