4

curl has an option to directly save file and header data on disk:

curl_setopt($curl_obj, CURLOPT_WRITEHEADER, $header_handle);
curl_setopt($curl_obj, CURLOPT_FILE, $file_handle);

Is there same ability in python-requests ?

rsk82
  • 28,217
  • 50
  • 150
  • 240
  • Have you checked out this? http://stackoverflow.com/questions/2776794/help-with-curl-in-python – Ryan G Jul 30 '13 at 17:24
  • Or this: http://stackoverflow.com/questions/14114729/save-a-file-using-the-python-requests-library – Udy Jul 30 '13 at 17:26

2 Answers2

7

As far as I know, requests does not provide a function that save content to a file.

import requests

with open('local-file', 'wb') as f:
    r = requests.get('url', stream=True)
    f.writelines(r.iter_content(1024))

See request.Response.iter_content documentation.

iter_content(chunk_size=1, decode_unicode=False)

Iterates over the response data. When stream=True is set on the request, this avoids reading the content at once into memory for large responses. The chunk size is the number of bytes it should read into memory. This is not necessarily the length of each item returned as decoding can take place.

falsetru
  • 357,413
  • 63
  • 732
  • 636
  • Does that mean if I request a big file it all gets loaded into memory if the medium doesn't support fast enough writing ? – rsk82 Jul 30 '13 at 17:40
  • @rsk82, No, this code does not read the content into memory at once. See added content. – falsetru Jul 30 '13 at 17:46
0

if you're saving something that is not a textfile, don't use f.writelines(). Instead use one of these:

import requests
try:
    r = requests.get(chosen, stream=True)
except Exception as E:
    print(E)
    # handle exceptions here.
# both methods work here...
with open(filepath, 'wb') as handle:
    for block in r.iter_content(1024):
        handle.write(block)
# or...
import shutil                    
with open(filepath, 'wb') as handle:
    shutil.copyfileobj(r.raw, handle)

shutil is much more flexible for dealing with missing folders or recursive file copying, etc. And it allows you to save the raw data from requests without worrying about blocksize and all that.

Marc Maxmeister
  • 4,191
  • 4
  • 40
  • 54
  • Using `r.raw` in this case is incorrect, as [it will not decode the `Transfer-Encoding`](https://requests.readthedocs.io/en/latest/user/quickstart/?highlight=download#raw-response-content:~:text=it%20does%20not%20transform%20the%20response%20content). – Kevin M Granger Apr 21 '23 at 14:30
  • Also, `writelines`, despite its name, does not add line endings. You can just think of it as a convenience function over an iterator. – Kevin M Granger Apr 21 '23 at 14:31