Flaky file deletion under Windows 7?

Question

I have a Python test suite that creates and deletes many temporary files. Under Windows 7, the shutil.rmtree operations sometimes fail (<1% of the time). The failure is apparently random, not always on the same files, not always in the same way, but it's always on rmtree operations. It seems to be some kind of timing issue. It is also reminiscent of Windows 7's increased vigilance about permissions and administrator rights, but there are no permission issues here (since the code had just created the files), and there are no administrator rights in the mix.

It also looks like a timing issue between two threads or processes, but there is no concurrency here either.

Two examples of (partial) stack traces:

  File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean
    shutil.rmtree(cleandir)
  File "c:\python23\lib\shutil.py", line 142, in rmtree
    raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg)
WindowsError: [Errno 5] Access is denied removing xml_1

  File "C:\ned\coverage\trunk\test\test_farm.py", line 298, in clean
    shutil.rmtree(cleandir)
  File "c:\python23\lib\shutil.py", line 142, in rmtree
    raise exc[0], (exc[1][0], exc[1][1] + ' removing '+arg)
WindowsError: [Errno 3] The system cannot find the path specified removing out

On Windows XP, it never failed. On Windows 7, it fails like this, across a few different Python versions (2.3-2.6, not sure about 3.1).

Anyone seen anything like this and have a solution? The code itself is on bitbucket for the truly industrious.

score 5 · Answer 1 · answered Jan 10 '10 at 15:37

5

It's a long shot, but are you running anything that scans directories in the background? I'm thinking antivirus/backup (maybe Windows 7 has something like that built in? I don't know). I have experienced occasional glitches when deleting/moving files from the TSVNCache.exe process that TortoiseSVN starts -- seems it watches directories for changes, then presumably opens them for scanning the files.

answered Jan 10 '10 at 15:37

j_random_hacker

50,331
10
105
169

I like this as a theory of why it's happening. – Ned Batchelder Jan 10 '10 at 17:01
In a comment on his post, Ryan Ginstrom mentions search indexing -- that sounds like another likely candidate to me. – j_random_hacker Jan 15 '10 at 01:46

Dave Markle · Answer 2 · 2011-04-03T12:34:08.743

1

My guess is that you should check up on the code that creates the file, and make SURE they are closed explicitly before moving on to delete it. If nothing is obvious there in the code, download a copy of Process Monitor and watch what's happening on the file system there. This tool will give you the exact error code coming from Windows, and should shed some light on the situation.

edited Apr 03 '11 at 12:34

answered Jan 10 '10 at 15:04

Dave Markle

95,573
20
147
170

That's a good idea, I'll look into it, but how would it explain the "cannot find the path specified" errors? Those seem to be rmtree trying to remove the tree *after* it had already been deleted? – Ned Batchelder Jan 10 '10 at 15:09
1

Aren't you thinking of [Process Monitor](http://technet.microsoft.com/en-us/sysinternals/bb896645), not Process Explorer? – Baffe Boyois Apr 02 '11 at 20:11

score 1 · Accepted Answer · edited May 23 '17 at 12:01

1

We had similar problems with shutil.rmtree on Windows, particularly looking like your first stack trace. We solved it by using an exception handler with rmtree. See this answer for details.

edited May 23 '17 at 12:01

Community

1
1

answered Jan 10 '10 at 15:13

ThomasH

22,276
13
61
62

Oddly, if I simply provide an onerror function that retries the operations, the problem seems to stop. – Ned Batchelder Jan 10 '10 at 15:33
Which might support your idea of a timing issue. – ThomasH Jan 10 '10 at 17:13

Ryan Ginstrom · Answer 4 · 2010-01-10T17:06:34.210

1

Just a thought, but if the test behavior (creating and deleting lots of temp files) isn't typical of what the app actually does, maybe you could move those test file operations to (c)StringIO, and keep a suite of function tests that exercises your app's actual file creation/deletion behavior.

That way, you can make sure that your app behaves correctly, without introducing extra complexity not related to the app.

edited Jan 10 '10 at 17:06

answered Jan 10 '10 at 16:03

Ryan Ginstrom

13,915
5
45
60

This is exactly the right thinking for improving the testability of code. In my case, the files and directories are important to the test, though. – Ned Batchelder Jan 10 '10 at 17:02
1

Right. In that case, make sure you turn off search indexing, "shadow copy," etc. I've also found Windows 7 to be very picky about deleting files from under other processes. – Ryan Ginstrom Jan 10 '10 at 17:17

score 0 · Answer 5 · answered Aug 31 '13 at 01:34

That "The system cannot find the path specified:" error will appear intermittently if the path is too long for Windows (260 chars). Automated tasks often create folder hierarchies using relative references that produce fully qualified paths longer than 260 characters. Any script that attempts to delete those folders using fully qualified paths will fail.

I built a quick workaround that used relative path references, but don't have generic code solution to offer, just a warning that the excellent answers provided might not help you.

score 0 · Answer 6 · answered May 17 '18 at 02:27

I met same problem with shutil.rmtree command and this issue maybe cause by special file name.(Ex:Other country language:леме / Ö)

Please use the following format to delete the directory which you want:

import shutil

shutil.rmtree(os.path.join("<folder_name>").decode('ascii'))

Enjoy it !

Flaky file deletion under Windows 7?

6 Answers6