0

I have been building a scraper which works well but requires me to manually expand the hidden data before it can successfully scrape it. I have checked the webpage source code and the data lies in 3x different accordion containers.

There is a table header banner which is clickable and contains a number of header elements that are also scraped. I have tried every possible xpath within the header that may contain the clickable element along with code like this:

driver.find_element_by_xpath('//*[@id="income-statement-accordion"]/thead/tr/th[1]').click

but no possible xpath combination expands the table. When I look at the source code, the only bit of source code that appears to change when I click the header is as follows

<table _ngcontent-ng-lseg-c34="" class="full-width income-statement swipable-table accordion-container" id="income-statement-accordion"> == $0

changes to this (I'm not sure the importance of == $0?):

<table _ngcontent-ng-lseg-c34="" class="full-width income-statement swipable-table accordion-container active-accordion" id="income-statement-accordion">

I have viewed a similar question here: Open an accordion with selenium in python. However, attempting a similar take on this does not do anything, although does admittedly submit without error and returns none:

driver.execute_script("document.getElementById('income-statement-accordion').class='full-width income-statement swipable-table accordion-container active-accordion';")

What am I missing?

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Tammboy
  • 321
  • 4
  • 14

1 Answers1

0

This seems to work

elements = driver.find_elements_by_xpath("//span[contains(@class, 'accordion-toggler')]")
for element in elements:
    driver.execute_script("arguments[0].click();", element)

This will expand all accordion sections.

If you just want to target individual sections

element = driver.find_element_by_xpath("//table[@id='income-statement-accordion']//span[contains(@class, 'accordion-toggler')]")
driver.execute_script("arguments[0].click();", element)

We need to use javascript to click the elements because selenium does not classify them as "interactable".

scilence
  • 1,069
  • 1
  • 5
  • 10