11

Is there a way to register a file so that it is deleted when Python exits, regardless of how it exits? I am using long-lived temporary files and want to ensure they are cleaned up.

The file must have a filename and it's original handle should be closed as soon as possible -- there will be thousands of these created and I need to ensure they exist only as normal files.

edA-qa mort-ora-y
  • 30,295
  • 39
  • 137
  • 267
  • I'm not quite sure that this makes any sense. What are you using these temporary files for? Also, what do you mean by "exist only as normal files" – wdh Jan 21 '14 at 12:23
  • Why are you (seemingly) passing data between various parts of your program using files? – sapi Jan 21 '14 at 12:27
  • 1
    @sapi: I use a temp dir for all sorts of external code or processes that require multiple files to be created, but I want to clean up. There are use cases that make sense, in any case. – Martijn Pieters Jan 21 '14 at 12:31
  • 1
    This is a great question, and a common one. I'd suggest it not be closed for "lack of focus". There is no problem with the clarity of the question, except that we would like to offer an answer besides "no". The truth is there is not an easy way to do what the OP would like to do that meets the criteria "regardless of how [Python] exits". There are other solutions that are inelegant but might fit. One is to have a parent process make a temporary directory in /tmp for the files, and to clean it up. Or to have subsequent runs of the job identify and clean up orphans from a past run. – Scott Smith Mar 17 '21 at 18:19

2 Answers2

17

Use the tempfile module; it creates temporary files that auto-delete.

From the tempfile.NamedTemporaryFile() documentation:

If delete is true (the default), the file is deleted as soon as it is closed.

You can use such a file object as a context manager to have it closed automatically when the code block exits, or you leave it to be closed when the interpreter exits.

The alternative is to create a dedicated temporary directory, with tempdir.mkdtemp(), and use shutil.rmtree() to delete the whole directory when your program completes.

Preferably, you do the latter with another context manager:

import shutil
import sys
import tempfile

from contextlib import contextmanager


@contextmanager
def tempdir():
    path = tempfile.mkdtemp()
    try:
        yield path
    finally:
        try:
            shutil.rmtree(path)
        except IOError:
            sys.stderr.write('Failed to clean up temp dir {}'.format(path))

and use this as:

with tempdir() as base_dir:
    # main program storing new files in base_dir

# directory cleaned up here

You could do this with a atexit hook function, but a context manager is a much cleaner approach.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • I am using that function, but I set "delete=False" since I don't want to delete the file on close. I need the name to exist beyond that point to be used in another part of the program. – edA-qa mort-ora-y Jan 21 '14 at 12:13
  • 1
    @edA-qamort-ora-y: You don't need to close it then. – Martijn Pieters Jan 21 '14 at 12:15
  • I've clarified the question, but essentially I can't keep the file open since there will be many of them. – edA-qa mort-ora-y Jan 21 '14 at 12:18
  • The contextmanager is an interesting approach. I can adjust my code to make it fit that format. `atexit` is also an option (I was unaware it existed in Python), but of course there are cases in which it won't be called (uncommon though). – edA-qa mort-ora-y Jan 21 '14 at 13:43
  • Is there any reason `yield path` is wrapped in a try/finally block? Is it possible for an exception to be thrown within that block? – logworthy Oct 09 '19 at 06:01
  • 1
    @logworthy yes, because *any exception raised in the `with` block* is propagated to the `yield` expression, by the `@contextmanager` implementation. – Martijn Pieters Oct 09 '19 at 06:42
2

You might be able to use the atexit library but this will not catch functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called. But I don't think anything in pure python can handle those cases.

You might also be interested in this question

Community
  • 1
  • 1
wdh
  • 1,612
  • 12
  • 16