0

Problem:

1) Need a python library which accepts path of file instead of opening the file(as it can be huge) to make a PUT request. 2) Also want that library to automatically do the redirect if exists(Requests library does that)

The shell command for this is curl -T

I've tried requests, it does redirect but needs to open file to send the file. Same is the case for pyCurl. The only alternative left is running a shell command in the python code.

ksn varma
  • 1
  • 1
  • When you say open the file, you mean open and read all of its contents into memory, right? Perhaps this might help you - https://stackoverflow.com/questions/29775247/requests-how-to-stream-upload-partial-file – GordonAitchJay Oct 03 '19 at 06:17
  • The pycurl example [file_upload.py](https://github.com/pycurl/pycurl/blob/master/examples/file_upload.py) does an upload in a streaming manner. Similar to `curl -T`. – Daniel Stenberg Oct 03 '19 at 06:25
  • @GordonAitchJay I didnt want to open the file, instead just want to give the file path and make a request, like `curl -T` which takes only the path – ksn varma Oct 03 '19 at 06:54
  • @DanielStenberg I can see, `open(filename, 'rb')` in the url u have given. Wanted to avoid that – ksn varma Oct 03 '19 at 06:55
  • even `curl -T` to send file it has to open it and read it to send data to socket/connection but you don't see this. Computers can't send file without opening it and reading it to memory (whole or piece by piece) – furas Oct 03 '19 at 07:33
  • You *can't* avoid opening the file if you want to get to the contents of it. You want to avoid loading the entire contents into memory first and *then* sending it but you can't avoid opening it. – Daniel Stenberg Oct 03 '19 at 09:46
  • @furas Didn't knew that. Thank you. Let me know if i have to delete this question. – ksn varma Oct 03 '19 at 10:34

1 Answers1

0

In pycurl, both READDATA and READFUNCTION should stream the data from the open file. As pointed out in the comments, opening the file is required to do anything with its contents.

In general you can verify that your program is streaming data rather than reading the file in memory by monitoring its memory usage while it is executing and providing it with a large input file (e.g. > 100 MB).

D. SM
  • 13,584
  • 3
  • 12
  • 21