How to upload a bunch of files with Python requests module

Question

When I using the requests module to upload at least 400 files with one request, I got the Too many open files error on macOS system.

I have tried ulimit -n 20000.

Also checked:

sysctl kern.maxfiles got 98304
sysctl kern.maxfilesperproc got 49152

However, it didn't work.

Here is my code:

import os
import requests


url = 'http://127.0.0.1:8000/api/upload'
file_path = '/Users/BonJu/Projects/downloads'
file_list = os.listdir(file_path)
files = []
for file in file_list:
    try:
        source = open(os.path.join(file_path, file), 'rb')
        files.append(('file', source))

    except Exception as e:
        print('File: %s, Error: %s' % (file, e.__str__()))
        continue

response = requests.post(url=url, data={'uploader': 'admin'}, files=files)

Terminal result:

File: test_252.docx, Error: [Errno 24] Too many open files: '/Users/BonJu/Projects/downloads/test_252.docx'
File: test_253.docx, Error: [Errno 24] Too many open files: '/Users/BonJu/Projects/downloads/test_253.docx'
File: test_254.docx, Error: [Errno 24] Too many open files: '/Users/BonJu/Projects/downloads/test_254.docx'
...
File: test_418.docx, Error: [Errno 24] Too many open files: '/Users/BonJu/Projects/downloads/test_418.docx'

Because it is an API server which links the sent log files to the issue I need all the files to be sent in one request, otherwise the manager will get several mails and won't be able to address the issues.

Is there any solution for this situation?

Solution

I finally adjusted my API to save a temp file for storing the upload logs and pass a status parameter to control the final output.

my code:

payload = {
    'status': 'finish',
    'uploader': 'admin'
}
response = requests.post(url=url, data=payload, files=files)

API:

@api_view(['post'])
def upload(request, debug, api_version):
    status = request.POST.get('status')
    file_list = request.FILES.getlist('file')

    if status == 'finish':
        # open the temp file and insert the last logs then output
        
    else:
        # create/insert the logs and save to a temp file

    return Response({'status': status, 'files': file_list})

Does this answer your question? [Why should I close files in Python?](https://stackoverflow.com/questions/25070854/why-should-i-close-files-in-python) — mkrieger1, Jul 13 '20 at 07:17
If you need to pass the file handles to `requests.post` and cannot close them before sending, you need to make multiple requests with fewer files. — mkrieger1, Jul 13 '20 at 07:23
@mkrieger1 OK~ I think it might be the best way for my situation. Thanks. — bonju0102, Jul 13 '20 at 07:31

score 1 · Answer 1 · answered Jul 13 '20 at 07:15

1

I would suggest make a single tar file from all of them.

or closing the files after reading their content (instead of leaving their file descriptor open until you post the request

answered Jul 13 '20 at 07:15

Mogi

596
1
6
18

I will try the `tar` way later. For the closing files tip, I tried and got `read of close file` after all. Thanks for your suggestion. – bonju0102 Jul 13 '20 at 07:26

How to upload a bunch of files with Python requests module

Solution

1 Answers1