0

How would one go about scraping a list from a < ul> tag if the list is only loaded in the div after a click is given on the original website?

I have tried using Selenium to automate the click, but because it doesn't change the https address of the website I am unsure how to get at this loaded-in data that otherwise returns < ul>< /ul> with Beautiful soup. The end goal is to get all of the data inside around 400 < li> tags inside of this dynamically loaded list, which is not seen in the inspect tool until after a button is clicked.

The website is: https://www.ibiscycles.com/test-ride/find-a-dealer

After manually clicking the search button the list of all applicable locations is loaded under this < div> name: This screenshot shows the breakdown using inspect

Thank you in advance!

bee13
  • 1
  • 2
  • 1
    What website? What code did you run? What output are you looking for (include the html element in question)? Minimal reproducible example is crucial here. – Arundeep Chohan Jun 24 '22 at 23:02
  • transfer the webdriver page_source to bs4. `BeautifulSoup(browser.page_source, 'lxml')` – QHarr Jun 25 '22 at 03:52

1 Answers1

0

Why can't you click the button? Wait for the ul to be clickable before selecting it. This should allow you to grab the ul after you've clicked on the button that loads it in

A B
  • 87
  • 1
  • 7