1

I have an observation about memory leak:

  • If we try to POST files to Django view(I use DRF's APIView), Django will not release memory even a response has been made. The memory will keep rising if you keep uploading files even if you do nothing in views.py. (I have tried uploading 100 files concurrently calling the same API). I know that if an uploaded file is smaller than 2.5 megabytes, Django will hold the entire contents of the upload in memory. So I am wondering this could be an issue for a possible memory leak because it will never release the memory.

PS: Even if I set FILE_UPLOAD_MAX_MEMORY_SIZE=0, which means use disk to store the files instead of using memory, I can still see a memory rise at the background.

I am using:

  • Python3.6/Django2.1, Gunicorn
  • DEBUG = False

Does anyone know how to solve this issue ? Thanks a lot.

Samuel
  • 11
  • 4

2 Answers2

1

It is impossible to tell what is causing the problem but here is a list of things you can try:

  • Ensure you are not using DEBUG=TRUE
  • Try upgrading your Python/Django versions if possible
  • What WSGI server are you using? (hopefully not the development server). Upgrade to the newest version or try a different one and compare (for example if you are using gunicorn try uwsgi or the opposite)
  • Not a solution but a workouround: gunicorn has a max_requests setting and you specify the number of requests before restarting the worker processes. Most servers have such a setting. Restarting will free memory if its leaking.

If the issue persists you may need to trace the cause with a tool like tracemalloc.

zxzak
  • 8,985
  • 4
  • 27
  • 25
  • Thanks. 1. DEBUG=False 2. I am using Python3.6/Django2.1 3. Gunicorn – Samuel Sep 25 '19 at 20:37
  • BTW if I read a file using file.read(), it seems like Python won't release memory either. – Samuel Sep 25 '19 at 20:46
  • Not sure where to start.. Let's say we create a POST API for file upload using DRF, you can try POST a 2MB file through this API and see if the memory goes up. If this is not obvious, try POST 100 or even 200 concurrently, then you will see. – Samuel Sep 25 '19 at 21:24
  • Are you closing the file after reading (by using a context manager or explicitly)? – zxzak Sep 25 '19 at 21:32
  • There is no way to "close" it. It comes from upload files, which is in the memory. file = request.data.get('file'), or file = request.FILES['file'] – Samuel Sep 26 '19 at 00:16
0

It turns out to be the problem of Gunicorn. Once setting the --thread, it seems like Gunicorn somehow won't release memory until all threads were used(Not sure if it is the correct term).

This could be helpful: Gunicorn Workers and Threads

Samuel
  • 11
  • 4