How to clean up temporary file used with send_file?

Question

I'm currently developing a server side json interface where several temporary files are manipulating during requests.

My current solution for cleaning up these files at the end of the request looks like this:

@app.route("/method",methods=['POST'])
def api_entry():
    with ObjectThatCreatesTemporaryFiles() as object:
        object.createTemporaryFiles()
        return "blabalbal"

In this case, the cleanup takes lace in object.__exit__()

However in a few cases I need to return a temporary files to the client, in which case the code looks like this:

@app.route("/method",methods=['POST'])
def api_entry():
    with ObjectThatCreatesTemporaryFiles() as object:
        object.createTemporaryFiles()
        return send_file(object.somePath)

This currently does not work, because when I the cleanup takes place flask is in the process of reading the file and sending it to the client. ¨ How can I solve this?

Edit: I Forgot to mention that the files are located in temporary directories.

Answer 1

The method I've used is to use weak-references to delete the file once the response has been completed.

import shutil
import tempfile
import weakref

class FileRemover(object):
    def __init__(self):
        self.weak_references = dict()  # weak_ref -> filepath to remove

    def cleanup_once_done(self, response, filepath):
        wr = weakref.ref(response, self._do_cleanup)
        self.weak_references[wr] = filepath

    def _do_cleanup(self, wr):
        filepath = self.weak_references[wr]
        print('Deleting %s' % filepath)
        shutil.rmtree(filepath, ignore_errors=True)

file_remover = FileRemover()

And in the flask call I had:

@app.route('/method')
def get_some_data_as_a_file():
    tempdir = tempfile.mkdtemp()
    filepath = make_the_data(dir_to_put_file_in=tempdir)
    resp = send_file(filepath)
    file_remover.cleanup_once_done(resp, tempdir)
    return resp

This is quite general and as an approach has worked across three different python web frameworks that I've used.

Answer 2

If you are using Flask 0.9 or greater you can use the after_this_request decorator:

@app.route("/method",methods=['POST'])
def api_entry():
    tempcreator = ObjectThatCreatesTemporaryFiles():
    tempcreator.createTemporaryFiles()

    @after_this_request
    def cleanup(response):
        tempcreator.__exit__()
        return response

    return send_file(tempcreator.somePath)

EDIT

Since that doesn't work, you could try using cStringIO instead (this assumes that your files are small enough to fit in memory):

@app.route("/method", methods=["POST"])
def api_entry():
    file_data = dataObject.createFileData()
    # Simplest `createFileData` method:  
    # return cStringIO.StringIO("some\ndata")
    return send_file(file_data,
                        as_attachment=True,
                        mimetype="text/plain",
                        attachment_filename="somefile.txt")

Alternately, you could create the temporary files as you do now, but not depend on your application to delete them. Instead, set up a cron job (or a Scheduled Task if you are running on Windows) to run every hour or so and delete files in your temporary directory that were created more than half an hour before.

Answer 3

I have two solutions.

---

The first solution is to delete the file in the __exit__ method, but not close it. That way, the file-object is still accessible, and you can pass it to send_file.

This will only work if you do not use X-Sendfile, because it uses the filename.

---

The second solution is to rely on the garbage collector. You can pass to send_file a file-object that will clean the file on deletion (__del__ method). That way, the file is only deleted when the file-object is deleted from python. You can use TemporaryFile for that, if you don't already.

Answer 4

It's a bit late, but this is what I did using madjar's suggestions (in case anyone else comes across this). This is a little helper function that I use (it takes a PyExcelerate Workbook object as parameter) which you could adapt to your case. Just change the way you create/build your tempfile.TemporaryFile and you're set! Tested on Windows 8.1 and Ubuntu 12.04.

def xlsx_to_response(wb, filename):
    f = tempfile.TemporaryFile()
    wb._save(f)
    f.seek(0)
    response = send_file(f, as_attachment=True, attachment_filename=filename,
                         add_etags=False)

    f.seek(0, os.SEEK_END)
    size = f.tell()
    f.seek(0)
    response.headers.extend({
        'Content-Length': size,
        'Cache-Control': 'no-cache'
    })
    return response

Answer 5

We can use tempfile.NamedTemporaryFile() which automatically deletes the file when send_file() closes it.

temp_csv_file = tempfile.NamedTemporaryFile(mode='w', encoding='utf-8', delete=True)
csv_writer = csv.writer(temp_csv_file)
csv_writer.writerow(["Name", "Age"])
for age in range(1,10):
    csv_writer.writerow(["alice", age])
return send_file(
    path_or_file=temp_csv_file.name,
    mimetype='text/csv',
    as_attachment=True,
    download_name='downloaded.csv'
    )

Answer 6

Windows / Linux / Mac compatible solution

I have tried weakrefs, flask built-in decorators and nothing worked.

The only think that worked in every system is to create a temporal file in memory using io.BytesIO

import os
import io
import tempfile
from multiprocessing import Process

import flask


def background_job(callback):
    task = Process(target=callback())
    task.start()

def send_temp_file(file_path: str, temp_dir: tempfile.TemporaryDirectory, remove_dir_after_send=True):
    with open(file_path, "rb") as f:
        content = io.BytesIO(f.read())
    response = flask.send_file(content,
                               as_attachment=True,
                               attachment_filename=os.path.split(file_path)[0])
    if remove_dir_after_send:
        background_job(temp_dir.cleanup)
    return response



app = flask.Flask(__name__)


@app.route("/serve_file/", methods=["GET"])
def serve_file():
    temp_dir = tempfile.TemporaryDirectory()
    file_path = os.path.join(temp_dir.name, "test.txt")
    with open(file_path, "w") as f:
        f.write("Hello World!")

    return send_temp_file(file_path, temp_dir)


if __name__ == "__main__":
    app.run(port=1337)