downloading a file from a webpage using python script without url , calling onClick function

Question

There is a webpage which have a link "Click to Download" Clicking which a file is downloaded . I can download this file manually by going to webpage and clicking on this link however I need to download this file via a python script .

If i see the source i can see the anchor tag is will run a js function

<a class="download-data-link1" onclick=" document.forms['dataform'].submit()" style="cursor:pointer; vertical-align: middle;">Download in csv</a>

But i dont know the url of csv file and i am looking for a way to download it via python .

I know we can download a file if we have url using httplib but couldnt understand how to get a file without url .

Tried few things like in header added 'Content-Disposition': 'attachment;filename="data.csv"'}

but it dosent seems to work . Any ideas ?

It has to python? Because links have a tendency (especially nowadays) to be added to the DOM later on, or to respond to certain events, triggered by the user, which makes a headless browser a better option here (IMHO). I'd look into PhantomJS or similar tools — Elias Van Ootegem, Jan 26 '15 at 09:13
Well even if its not python maybe something which i can invoke from python script . Just saw http://phantomjs.org/ as per your suggestion . Need to study it further — paarth batra, Jan 26 '15 at 09:18
`PhantomJS` can be automated through `selenium` - so it can be python. — alecxe, Jan 26 '15 at 09:19
@paarthbatra: [Selenium + chromedriver](https://github.com/EVODelavega/gp-scripts/tree/master/codeception) would be an option, too (link is to 2 scripts: get-drivers downloads what you need, `codeception-start` starts/stops selenium), you can then use that setup with a python script quite easily — Elias Van Ootegem, Jan 26 '15 at 09:26

score 3 · Answer 1 · edited Jun 30 '17 at 17:53

Thanks all for your answers but I want to add, how i implemented it.

First of all you can create a firefox profile. To do that:
Close all firefox browsers
go to cmd prompt and execute firefox.exe -P
create a profile and note down the name of the folder where new profile is created

You can set some options for your profile here, like - automatically download these kind of files from content etc.

Now Download selenium for python and use below code

import os
from selenium import webdriver

download_dir="D:\a"

fp = webdriver.FirefoxProfile(<profile directory here as in step 4>)
fp.set_preference("browser.download.dir", download_dir)
fp.set_preference("browser.download.manager.showWhenStarting",False)
fp.set_preference("browser.helperApps.neverAsk.saveToDisk", "application/octet-stream");

browser = webdriver.Firefox(firefox_profile=fp)
browser.get("http://pypi.python.org/pypi/selenium")
# you can use your url here 
browser.find_element_by_partial_link_text("selenium-2").click()
# Use your method to identify class or link text here
browser.close();

Hope this may help others :)

score 2 · Accepted Answer · edited May 23 '17 at 11:59

Two basic options can be applied here:

mimic the logic involved in the onclick() call - in your case, make the dataform form submission using requests, or mechanize

high-level approach - automate a real browser, headless (PhantomJS) or not, using selenium - find the link and click it:

from selenium import webdriver

driver = webdriver.PhantomJS()
driver.get('url here')

driver.find_element_by_class_name('download-data-link1').click()

Though, as far as I understand, clicking the link would trigger a "Download" browser dialog to appear - then PhantomJS is not an option since it doesn't support downloads. In case of Chrome or Firefox you would need to tweak browser capabilities to automatically download files without opening the popup, see:

downloading a file from a webpage using python script without url , calling onClick function

2 Answers2