0

I had a vps with 1 GB ram and i wanted send a big file (2GB ) by curl and python.

python gave MemoryLimit error but curl send file without problem and use very low memory.

How curl did it?

How i can send a very big file in a Http post request with use low memory?

Charles Duffy
  • 280,126
  • 43
  • 390
  • 441
yasser karimi
  • 329
  • 3
  • 13
  • Applications use virtual memory, they're not limited by the amount of RAM. – Barmar Aug 18 '21 at 15:05
  • @Barmar So you're saying curl actually did load all 2 GB into (virtual) memory at once and then sent them? – Kelly Bundy Aug 18 '21 at 15:06
  • @Carcigenicate If he's using a library to upload the file, the file reading is internal to the library. – Barmar Aug 18 '21 at 15:07
  • Why would you read the whole file into memory at once, instead of reading (and sending) a block at a time? – Charles Duffy Aug 18 '21 at 15:55
  • 1
    This question would be helped by a [mre] -- an example of your upload code, so answers can be written to use roughly the same tools (are you using `urllib`? `urllib2`? `requests`? something else?). – Charles Duffy Aug 18 '21 at 15:56

1 Answers1

1

You might have loaded all file data into memory and then tried to send it. This obviously results in MemoryLimit error.

You could try to use mmap module. A sample code would be as follows:

import requests
import mmap

with open("hello.txt", "r+b") as f:
    # memory-map the file, size 0 means whole file
    mm = mmap.mmap(f.fileno(), 0)
    r = requests.post(URL, data=mm)
    print(r.status_code, r.reason)
    mm.close()
Tail of Godzilla
  • 531
  • 1
  • 6
  • 20