0

I can't set profile with automatic download for CSV file when using selenium.

Among the others, I tried solutions presented in the following questions:

On my "regular firefox" I can set the rule to download it automatically, it works, however when geckodriver runs firefox, this rule/profile does not apply.

I have administrator rights. And As I read on another question, my orange bar in firefox is a result of controlling it by webdriver.

Here is sample of my code:

profile = webdriver.FirefoxProfile()
profile.set_preference("browser.download.panel.shown", False)
profile.set_preference("browser.helperApps.neverAsk.openFile","text/csv,application/vnd.ms-excel")
profile.set_preference("browser.helperApps.neverAsk.saveToDisk", "CSV File,text/csv,application/vnd.ms-excel")
profile.set_preference("browser.download.folderList", 2);
profile.set_preference("browser.download.dir", "F:\\Delete")
driver=webdriver.Firefox(firefox_profile=profile)

1. Could you guys help me understand what am I mising?

2. How can I download CSV files automatically?

furas
  • 134,197
  • 12
  • 106
  • 148
Hronic
  • 155
  • 1
  • 1
  • 7
  • these solutions are 6 and 9 years old. In this times Selenium or rather browsers could change everything in settings. – furas Dec 31 '20 at 18:20
  • I tested code on some CSV - and I had to use `application/octet-stream` with `text/csv,application/vnd.ms-excel` because sometimes servers may send file with different type. – furas Dec 31 '20 at 18:43

2 Answers2

1

I have been using the below logic and working for me in Java, you can easily port this to python.

FirefoxProfile profile = new FirefoxProfile();

        // set the download folder directory
        profile.setPreference("browser.download.dir", this.getDownloadFolderPath());

        // the last folder specified for a download
        profile.setPreference("browser.download.folderList", 2);

        // hide Download Manager window when a download begins
        profile.setPreference("browser.download.manager.showWhenStarting", false);

        /**
         This is the most important setting that will make sure the pdf is downloaded
         without any prompt
         */
        profile.setPreference("pdfjs.disabled", true);

        profile.setPreference("pref.downloads.disable_button.edit_actions", false);
        profile.setPreference("media.navigator.permission.disabled", true);

        // A comma-separated list of MIME types to save to disk without asking what to
        // use to open the file.
        profile.setPreference("browser.helperApps.neverAsk.saveToDisk",
                "application/pdf,application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/zip,text/csv,text/plain,application/x-msexcel,application/excel,application/x-excel,application/vnd.ms-excel,image/png,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;octet/stream");

        // A comma-separated list of MIME types to open directly without asking for
        // confirmation.
        profile.setPreference("browser.helperApps.neverAsk.openFile",
                "application/pdf,application/msword, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/zip,text/csv,text/plain,application/x-msexcel,application/excel,application/x-excel,application/vnd.ms-excel,image/png,application/vnd.openxmlformats-officedocument.spreadsheetml.sheet;octet/stream");

        // Do not ask what to do with an unknown MIME type
        profile.setPreference("browser.helperApps.alwaysAsk.force", false);

        // Leave the window in the background when starting a download (Default Setting
        // is false)
        profile.setPreference("browser.download.manager.focusWhenStarting", false);

        // popup window at bottom right corner of the screen will not appear once all
        // downloads are finished.
        profile.setPreference("browser.download.manager.showAlertOnComplete", true);

        // Close the Download Manager when all downloads are complete
        profile.setPreference("browser.download.manager.closeWhenDone", true);

        FirefoxOptions options = new FirefoxOptions();
        options.setProfile(profile);

Make sure to consume the options while creating the driver instance. And let me know if you are still facing the issue.

supputuri
  • 13,644
  • 2
  • 21
  • 39
  • Thank you, I checked it and worked. The problem was on my side, MIME which I was missing was: `binary/octet-stream`. – Hronic Jan 07 '21 at 14:53
1

I tested your code on some CSV and I had to use application/octet-stream because sometimes servers may send file with different type.

As I remeber application/octet-stream was popular method on web pages to force downloading - especially for PDF - because some users may have settings in browser which display PDF with built-in viewer instead of downloading.

Minimal working code with example CSV

from selenium import webdriver

url = 'https://data.europa.eu/euodp/pl/data/dataset/covid-19-coronavirus-data'

profile = webdriver.FirefoxProfile()
profile.set_preference("browser.download.panel.shown", False)
profile.set_preference("browser.helperApps.neverAsk.openFile","text/csv,application/vnd.ms-excel,application/octet-stream")
profile.set_preference("browser.helperApps.neverAsk.saveToDisk", "CSV File,text/csv,application/vnd.ms-excel")
profile.set_preference("browser.download.folderList", 2);
profile.set_preference("browser.download.dir", "/home/furas/")
driver = webdriver.Firefox(firefox_profile=profile)

driver.get(url)

item = driver.find_element_by_xpath('//div[@id="dataset-resources"]//li[2]//a')
print('text:', item.text)
print('href:', item.get_attribute('href'))
item.click()

EDIT: For other users: it needed also binary/octet-stream

furas
  • 134,197
  • 12
  • 106
  • 148
  • Hello, thank you for the answer. I tried out your code, however I am still prompted by Firefox to open/save file. There is no error message, just regular window "where to save". – Hronic Jan 07 '21 at 13:19
  • maybe you need other option like `profile.set_preference("browser.helperApps.alwaysAsk.force", False)` – furas Jan 07 '21 at 13:33
  • I inserted the code before line: `profile.set_preference("browser.download.folderList", 2);` The same effect - window pop out With your example (the same link etc), I did not download the file, but firefox successfully defined it as "auto-download", because when i clicked hyperlink, file was automatically downloaded to Download location. Link generated: https://opendata.ecdc.europa.eu/covid19/casedistribution/csv Is it relative that I download the file after simulation of clicking "download" button? – Hronic Jan 07 '21 at 13:44
  • I don't know - it downloads me withiout asking when I use `item.click()` or `driver.get('https://opendata.ecdc.europa.eu/covid19/casedistribution/csv')` – furas Jan 07 '21 at 13:56
  • or maybe it depends on system or firefox version - I use Linux Mint 20 and Firefox 84.0.1 (64-bit) – furas Jan 07 '21 at 13:58
  • Windows, Firefox 84.0.2 (32-bit) I'm sorry, I copied your code once again, run it and it worked (I mean it downloaded the file without asking). The problem is the format, let's say it seems it has no extension - I can open it with notepad. What would be the difference between "my file" and yours (because code is the same) - it is like my "csv" is not "csv" that's why it prompts for downloading. – Hronic Jan 07 '21 at 14:14
  • I changed `[3]` to `[2]` because it started geting me next link with `xml` file - but it downloaded me `xml` also without asking. But maybe it sends it with different MIME then `text/csv,application/vnd.ms-excel,application/octet-stream` and your browser had to ask what to do. – furas Jan 07 '21 at 14:25
  • Is there any way I can check what kind of MIME I am encountering? Some sort of options? – Hronic Jan 07 '21 at 14:39
  • 1
    if you use DevTools in Firefox/Chrome (tab: Network) then you can see all requests send to browser - and then you can see what headers it was used - one of them should be: `Content-Type` – furas Jan 07 '21 at 14:42
  • 1
    Man, it took me a lot of time, but thanks to your help I solved it. As you wrote above, I went into DevTools, found 'Content-Type` after manual download and it was `binary/octet-stream` -> I put it into line `profile.set_preference("browser.helperApps.neverAsk.saveToDisk", "CSV File,text/csv,application/vnd.ms-excel,application/octet-stream, binary/octet-stream")` and it works perfectly. Once again, thanks a lot! – Hronic Jan 07 '21 at 14:50
  • nice - I add this information to answer. It will be more visible. – furas Jan 07 '21 at 14:58