python: How can I download data from the webpage where the link is hidden by the download button?

Question

Suppose I want to download data here: http://www.dce.com.cn/publicweb/quotesdata/memberDealPosiQuotes.html

When click the button shown below, I got a .csv file:

I want to do this automatically using python where I can specify the date etc.

I find here that one can use pandas pd.read_csv to read data from webpage, but first one need to get the right url. However in my case I don't know what the url is.

Besides, I also want to specify the date and the contract etc. myself.

Before asking, I actually tried to the dev tool, I still can't see the url, and I don't know how to make it programatic.

`javascript:exportData('excel')`, you can click on that button, programmatic. — bhansa, Sep 13 '17 at 12:40
That could be achieved programmatically as @bhansa said. Selenium python bindings will do it for you. http://selenium-python.readthedocs.io/getting-started.html — Ahmed Dhanani, Sep 13 '17 at 12:51
@MrPyCharm How can I send a click not some text as in the example? — an offer can't refuse, Sep 13 '17 at 13:24
https://stackoverflow.com/questions/18439851/downloading-file-using-selenium — Ahmed Dhanani, Sep 13 '17 at 13:25

Håken Lid · Accepted Answer · 2017-11-03T13:44:30.187

The javascript exportData('excel') results in a form that is submitted. By using Chrome devtools and the Network panel, you can figure out the headers and the post data used, and then write a python script to submit an identical http request.

import requests
url = 'http://www.dce.com.cn/publicweb/quotesdata/exportMemberDealPosiQuotesData.html'
formdata = {
    'memberDealPosiQuotes.variety':'a',
    'memberDealPosiQuotes.trade_type':0,
    'contract.contract_id':'all',
    'contract.variety_id':'a',
    'exportFlag':'excel',
}
response = requests.post(url, data=formdata)
filename = response.headers.get('Content-Disposition').split('=')[-1]
with open(filename, 'wb') as fp:
    fp.write(response.content)

It's probably possible to find ways to modify the post data to fetch different data. Either by reverse engineering, by trial and error or by finding some documentation.

For example, you can include fields for year and date:

    'year':2017,
    'month':3,
    'day':20

python: How can I download data from the webpage where the link is hidden by the download button?

1 Answers1

Linked