Deleting large numbers of files in python

Question

I'm trying to remove all files found in a directory. The accepted answer to Delete Folder Contents in Python suggests getting a list of all files and calling "unlink" on them in a loop.

Suppose I have thousands of files on a network share, and want to tie up the directory for as short a time as possible.

Is it more efficient to delete them all using a shell command like rm -f /path/* or by using shutils.rmtree or some such?

If you move the directory to a new name (deleteme is good) then mkdir a new one, you can delete the old one in the background at your leisure. — nmichaels, Aug 04 '11 at 17:45
interesting @nmichaels. What about simply moving it to /dev/null ? — Dave, Aug 04 '11 at 17:47
That won't be any faster than deleting; moving with `os.rename` on the same filesystem is super fast. — agf, Aug 04 '11 at 17:52
@Dave: Interesting idea, but `mv: cannot overwrite non-directory `/dev/null' with directory `deleteme/'` — nmichaels, Aug 04 '11 at 17:53
If you can use Python 3.5, using os.scandir() in place of os.listdir() should be substantially faster. — Michael Scott Asato Cuthbert, Apr 02 '16 at 14:35

agf · Accepted Answer · 2011-08-04T20:30:36.303

10

If you actually want to delete the whole directory tree, shutils.rmtree should be faster than os.remove (which is the same as os.unlink). It also allows you to specify a callback function to handle errors.

The suggestion in the comment by @nmichaels is also good, you can os.rename the directory then make a new one in its place and use shutils.rmtree on the original, renamed directory.

edited Aug 04 '11 at 20:30

answered Aug 04 '11 at 17:47

agf

171,228
44
289
238

The question is about comparing python (rmtree is the best i think) to unix shell (rm -Rf ). – orzel Jun 11 '17 at 21:43
@orzel That's what he said, but I thought what he really meant was how to do it quickly, which `rmtree` does. He accepted my answer so we have to assume I addressed his intent. – agf Jun 12 '17 at 21:59
1

Might be. I am interested in the original question : which one is faster, python/rmtree or forking a 'rm -Rf' ? I was looking for tests/figures and arguments about this. – orzel Jun 13 '17 at 22:52

score 0 · Answer 2 · edited May 13 '19 at 18:14

0

I tried this solution and it seems to work well:

while os.path.exists(file_to_delete):
  os.remove(file_to_delete)

edited May 13 '19 at 18:14

Misantorp

2,606
1
10
18

answered May 13 '19 at 16:46

Benjamin Freoa

1

Deleting large numbers of files in python

2 Answers2