How to serve a created tempfile in django

Question

I have a remote storage project that when the user requests his file, the django server retrieves and stores the file locally (for some processing) as a temporary file and then serves it to the user with mod x-sendfile. I certainly want the tempfile to be deleted after it is served to the user.

The documentations state that NamedTemporaryFile delete argument if set to False leads to deletion of the file after that all the references are gone. But when the user is served the tempfile, it doesn't get deleted. If I set the delete=True in case of downloading I get the "The requested URL /ServeSegment/Test.jpg/ was not found on this server."

Here is a view to list the user files:

def file_profile(request):
    obj = MainFile.objects.filter(owner=request.user)
    context = {'title': 'welcome',
               'obj': obj
               }
    return render(request, 'ServeSegments.html', context=context)

This is the view which retrieves, stores temporarily and serve the requested file:

def ServeSegment(request, segmentID):    
    if request.method == 'GET':    
        url = 'http://192.168.43.7:8000/foo/'+str(segmentID)
        r = requests.get(url, stream=True)
        if r.status_code == 200:
            with tempfile.NamedTemporaryFile(dir=
        '/tmp/Files', mode='w+b') as f:
                for chunk in r.iter_content(1024):
                    f.write(chunk)        
            response = HttpResponse()
            response['Content-Disposition'] = 'attachment; segmentID={0}'.format(f.name)
            response['X-Sendfile'] = "{0}".format(f.name)
            return response
        else:
            return HttpResponse(str(segmentID))

I guess if I could manage to return the response inside with a statement and after that, the last chunk was written, it would work as I want, but I found no solution regarding how to determine if we are in the last loop (without being hackish).

What should I do the serve the tempfile and have it deleted right after?

Answer 1

Adding a generalized answer (based on Cyrbil's) that avoids using signals by doing the cleanup in a finally block.

While the directory entry is deleted by os.remove on the way out, the underlying file remains open until FileResponse closes it. You can check this by inspecting response._closable_objects[0].fileno() in the finally block with pdb, and checking open files with lsof in another terminal while it's paused.

It looks like it's important that you're on a Unix system if you're going to use this solution (see os.remove docs)

https://docs.python.org/3/library/os.html#os.remove

import os
import tempfile
from django.http import FileResponse

def my_view(request):
    try:
        tmp = tempfile.NamedTemporaryFile(delete=False)
        with open(tmp.name, 'w') as fi:
            # write to your tempfile, mode may vary
        response = FileResponse(open(tmp.name, 'rb'))
        return response
    finally:
        os.remove(tmp.name)

Answer 2

Any file created by tempfile will be deleted once the file handler is closed. In your case, when you exit the with statement. The delete=False argument prevent this behavior and let the deletion up to the application. You can delete the file after its been sent by registering a signal handler that will unlink the file once response is sent.

Your example does nothing on the file, so you might want to stream the content directly with StreamingHttpResponse or FileResponse. But as you said you "stores the file locally (for some processing)", I would suggest thinking on doing the processing without any temporary file created and only work with streams.

Answer 3

Disposable files The solution to the question is to not use with in the NamedTemporaryFile and handle exceptions. Currently your file is being deleted before your read. At the end return

f.seek(0)
return FileResponse(f, as_attachment=True, filename=f.name)

The temporary file will be closed when the read is complete and therefore deleted.

Non-disposable files

For those who stumble across do not have an automatically disposable file handle.

From the other answers, signals seemed to be a reasonable solution however passing data required altering protected members. I was unsure how supported it would be in the future. I also found that whp's solution did not work in the current version of Django. The most future-proof version I could come up with was monkey patching the file output so the file is deleted on close. Django closes the file handles at the end of sending the file and I can't see that changing.

def my_view(request):
  tmp = tempfile.NamedTemporaryFile(delete=False)

  try:
    # write file tmp (remember to close if re-opening)
    # after write close the file (if not closed)      

    stream_file = open(tmp.name, 'rb')

    # monkey patch the file
    original_close = stream_file.close

    def new_close():
        original_close()
        os.remove(tmp.name)

    stream_file.close = new_close

    # return the result
    return FileResponse(stream_file, as_attachment=True, filename='out.txt')
  except Exception:
      os.remove(output.name)
      raise