Python code gets Memoryerror in API Request

Question

Here is the python code snippet

with open(path, "rb") as sample:
payload = {

    'options': (None,json.dumps (options),'application/json'),'filename':(os.path.basename(path), sample,'application/octet-stream')
}


response = requests.request ("POST", url, headers=headers, files=payload, verify=False)

Files are usually 2GB. I need to upload but while uploading, I am getting a memory error.

 File "C:\Users\H3731736\PycharmProjects\website\venv\lib\site-packages\urllib3\filepost.py", line 92, in encode_multipart_formdata
body.write(b"\r\n")

Error:MemoryError

Please help me understand why this happening and how to solve this.

Could you tell exactly what is it that you are trying to achieve ? — Tanishq Vyas, Jun 02 '21 at 16:18
@TanishqVyas I am trying to upload the file on the portal, but as you can see I am getting memoryerror due to its ram gets full while reading the file in memory. Smaller files work fine but for larger files, it's showing an error. — HIMANSHU CHORSIYA, Jun 02 '21 at 16:24
@HIMANSHUCHORSIYA When you try to ready a whole file into a variable, the RAM gets full and that exactly is the problem. Maybe you should consider looking into websockets. That way you could send your file in chunks rather than in one go. Maybe that would help. — Tanishq Vyas, Jun 02 '21 at 16:28
@TanishqVyas reading in chunks will not help ,as i am sending the file directly via API using response — HIMANSHU CHORSIYA, Jun 02 '21 at 16:30

score 0 · Accepted Answer · answered Jun 02 '21 at 16:29

0

I tested this on a VM and found out that Python 32-bit version has necessary upper memory limits because of the address space. 64-bit has significantly more address space and switching to the 64bit version fixes the issue.

answered Jun 02 '21 at 16:29

Arib Muhtasim

147
2
8

Python code gets Memoryerror in API Request

1 Answers1