1

I'm using a python package, cdo, which heavily relies on tempfile for storing intermediate results. The created temporary files are quite large and when running bigger calculations, I've run into the problem that the /tmp directory got filled up and the script failed with a disk full error (we are talking about 10s to 100s of GB). I've found a workaround to the problem by creating a local folder, say $HOME/tmp and then doing

import tempfile
tempfile.tempdir='$HOME/tmp'

before importing the cdo module. While this works for me, it is somewhat cumbersome if I want also others to use my scripts. Therefore I was wondering, whether there would be a more elegant way to solve the problem by, e.g., telling tmpfile periodically to clear out all temporary files (usually this is only done once the script finishes). From my side this would be possible, because I am running a long loop, which produces one named file each iteration and all the temporary files created during that iteration would be discardable afterwards.

Thomas Kühn
  • 9,412
  • 3
  • 47
  • 63
  • Just use the tempfile module discussed in https://stackoverflow.com/questions/21257782/how-to-remove-file-when-program-exits It will auto-delete temp files when the file is closed. – MeterLongCat Jun 21 '17 at 10:57
  • @MeterLongCat As stated in the question, cdo uses `tempfile`, and I need to delete *before* the script finishes. Unfortunately I don't have control over the parameters that cdo passes to the `TemporaryFile()` instances upon creation. – Thomas Kühn Jun 21 '17 at 10:58

1 Answers1

2

as the examples show: you could use tempfile in a context manager:

with tempfile.TemporaryFile() as fp:
    fp.write(b'Hello world!')
    fp.seek(0)
    fp.read()

that way they are removed when the context exits.

...do you have that much control over how cdo uses tempfiles?

hiro protagonist
  • 44,693
  • 14
  • 86
  • 111