0

I am trying to scrape data from website with beautiful soup, but to scrape all content, I have to click button 

<button class="show-more">view all 102 items</button>

to load every item. I have heard that it could by done with selenium, but it means that i have to open browser with script, and then scrape the data. Are there any other ways to solve this problem.

Stephen Kennedy
  • 20,585
  • 22
  • 95
  • 108
Leczo
  • 11
  • 2

1 Answers1

1

You can use the same API endpoint the page does which returns all the info in json form. Set a records return count higher than the total expected number. I show parsing out the album titles/urls from the json. You can explore response here. You can find this endpoint in the browser network tab when refreshing the url you supplied.

import requests

data = {"fan_id":1812622,"older_than_token":"1557167238:2897209009:a::","count":1000}
r = requests.post('https://bandcamp.com/api/fancollection/1/wishlist_items', json = data).json()
details = [(item['album_title'], item['item_url']) for item in r['items']]
print(details)
QHarr
  • 83,427
  • 12
  • 54
  • 101