0

I want to scrap this page to automatically download all the files in it: https://spreadstreet.io/database/.

The trick is that the dropdown button () starts with a value of '10'. I want it to be set on 'all' instead.

But I don't find any of the 'div' related to this dropdown when I print the results in my console, despite seeing them in my browser!

I've followed this good answer: scraping a response from a selected option in dropdown list

But I don't understand where can I find those informations and therefor change them.

LeBruceWayne
  • 109
  • 1
  • 8

1 Answers1

1

Looks like your table load its data dynamically. BeautifulSoup itself is not gonna be enough. So you will need to use a tool like selenium.

You can also achieve your purpose with sending http request to source.

You can check your browser and check where it sends data and create request from it.

Here is the solution for your question.

from bs4 import BeautifulSoup
import requests
import json

url = "https://spreadstreet.io/wp-admin/admin-ajax.php"

querystring = {"action":"get_wdtable","table_id":"102"}

payload = "draw=4&columns%5B0%5D%5Bdata%5D=0&columns%5B0%5D%5Bname%5D=Cryptocurrency&columns%5B0%5D%5Bsearchable%5D=true&columns%5B0%5D%5Borderable%5D=true&columns%5B0%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B0%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B1%5D%5Bdata%5D=1&columns%5B1%5D%5Bname%5D=frequency&columns%5B1%5D%5Bsearchable%5D=true&columns%5B1%5D%5Borderable%5D=true&columns%5B1%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B1%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B2%5D%5Bdata%5D=2&columns%5B2%5D%5Bname%5D=Exchange&columns%5B2%5D%5Bsearchable%5D=true&columns%5B2%5D%5Borderable%5D=true&columns%5B2%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B2%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B3%5D%5Bdata%5D=3&columns%5B3%5D%5Bname%5D=database_code&columns%5B3%5D%5Bsearchable%5D=true&columns%5B3%5D%5Borderable%5D=true&columns%5B3%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B3%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B4%5D%5Bdata%5D=4&columns%5B4%5D%5Bname%5D=Description&columns%5B4%5D%5Bsearchable%5D=true&columns%5B4%5D%5Borderable%5D=true&columns%5B4%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B4%5D%5Bsearch%5D%5Bregex%5D=false&columns%5B5%5D%5Bdata%5D=5&columns%5B5%5D%5Bname%5D=Download&columns%5B5%5D%5Bsearchable%5D=true&columns%5B5%5D%5Borderable%5D=true&columns%5B5%5D%5Bsearch%5D%5Bvalue%5D=&columns%5B5%5D%5Bsearch%5D%5Bregex%5D=false&order%5B0%5D%5Bcolumn%5D=0&order%5B0%5D%5Bdir%5D=asc&start=0&length=-1"
headers = {
    'Accept': "text/html; charset=UTF-8",
    'Content-Type': "application/x-www-form-urlencoded"
    }

response = requests.request("POST", url, data=payload, headers=headers, params=querystring)

crpytos = json.loads(response.text)['data']

for crpyto in crpytos:
    name = crpyto[0]
    duration = crpyto[1]
    exchange = crpyto[2]
    download_link = BeautifulSoup(crpyto[5],'lxml').find('a')['href']
    print(name,duration,exchange,download_link)

This will give you the output of

.
.
.
.
Yacoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/YAC-BTC Exchange Rate.csv
Zcash daily Bitfinex https://spreadstreet.io?nc_download=1&path=Bitfinex/ZEC-BTC Exchange Rate.csv
Zcash daily Bitfinex https://spreadstreet.io?nc_download=1&path=Bitfinex/ZEC-USD Exchange Rate.csv
Zcash daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZEC-CNY Exchange Rate.csv
Zcash daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZEC-BTC Exchange Rate.csv
Zcash 5 min Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/5-Minute/USDT_ZEC 5-Minute.csv
Zcash 15 Min Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/15-Minute/USDT_ZEC 15-Minute.csv
Zcash 30 Min Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/30-Minute/USDT_ZEC 30-Minute.csv
Zcash 2 Hour Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/2-Hour/USDT_ZEC 2-Hour.csv
Zcash 4 Hour Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/4-Hour/USDT_ZEC 4-Hour.csv
Zcash daily Poloniex https://spreadstreet.io?nc_download=1&path=Poloniex/Daily/USDT_ZEC Daily.csv
Zcash Tick Bitfinex https://spreadstreet.io?nc_download=1&path=Bitfinex/Tick/ByCurrency/ZECBTC.csv
Zcash Tick Bitfinex https://spreadstreet.io?nc_download=1&path=Bitfinex/Tick/ByCurrency/ZECUSD.csv
ZcCoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZCC-CNY Exchange Rate.csv
ZcCoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZCC-BTC Exchange Rate.csv
Zetacoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZET-CNY Exchange Rate.csv
Zetacoin daily BTER https://spreadstreet.io?nc_download=1&path=BTER/ZET-BTC Exchange Rate.csv
Selçuk
  • 1,088
  • 1
  • 8
  • 12