Why does ReplaceFile fail with ERROR_SHARING_VIOLATION?

Question

While the documentation is vague, based on this question and comments and this answer, I expected that ReplaceFile called with the third argument (backup filename) should succeed even if there are handles to source and destination files open in other processes without FILE_SHARE_DELETE flag. It's supposed to be overcome the lock by changing just the file metadata (= directory entry), which is not controlled by the lock. (All three files are on the same disk drive, so changing metadata is enough to rename them.)

However, the code below fails with ERROR_SHARING_VIOLATION. This is not my use case, but just a demonstration of the failure. The use case is that I'm trying to rename files that are occasionally (and unpredictably) open in other processes on the system, such as antivirus or backup programs, which didn't bother to use FILE_SHARE_DELETE flag.

# python 3
import os
import ctypes

fname1 = 'test1.txt'
fname2 = 'test2.txt'
f1 = open(fname1, 'w')
f1.write(fname1)
f2 = open(fname2, 'w')
f2.write(fname2)

# tmp123 does not exist when the program is started
ctypes.windll.kernel32.ReplaceFileW(fname2, fname1, 'tmp123', 0, None, None) # 0
ctypes.GetLastError() # ERROR_SHARING_VIOLATION

# if we close file handles, it works as expected
f1.close()
f2.close()
ctypes.windll.kernel32.ReplaceFileW(fname2, fname1, 'tmp123', 0, None, None) # 1

Why?

same reason why you cant delete a file if it is open in another application one would guess. After writing, if you are done you should always just close the files. I think this is common in most languages. You open a file, you cant open it again for R/W/A priv unless you close the first reference to it. — Fallenreaper, Apr 24 '17 at 01:49
@Fallenreaper I edited the question to clarify what I'm trying to do. And I wouldn't be surprised with the outcome, except that the posts I linked in my question very clearly suggest that it should work fine. — max, Apr 24 '17 at 01:53
One of your posts says it is hard to resolve the issue because it is inherantly built into windows, the idea of race conditions against files. The other says that renaming will not delete the file and should not throw errors. Since it is a sharing error this would lead me to believe that even renaming it would cause an issue because another process still has it open. One option could be to just spin up some sort of sub process which will do a replace when the processes finish whatever they are doing. If it's required to continue, you will have to set up a wait, or find&kill the process — Fallenreaper, Apr 24 '17 at 02:09
@Fallenreaper yes, that's my backup approach. It did seem though that the 3-argument version of `ReplaceFile` was capable of overcoming the file lock problem that `MoveFileEx` and others could not. I guess it's not the case. — max, Apr 24 '17 at 02:13
yeah i am reading that. The point of his answer was that the user wanted to delete the file and couldnt. The answer was focused on the idea of the replace which just did the rename as you were seeing. It doesnt seem to show whether or not that line of code worked or not because it was selected as the best answer because it then would attempt to delete, or carry out the delete later. Maybe the concept of replace works, but not if it has an open process still. Did a google search and most said to find and kill the process and then rename, delete, etc. — Fallenreaper, Apr 24 '17 at 02:27
There is a distinction between your use case (transient open files, either using [opportunistic locking](https://blogs.msdn.microsoft.com/oldnewthing/20130415-00/?p=4663) or only opened for a brief period) and your sample code. Have you tried using ReplaceFile in the real code? — Harry Johnston, Apr 24 '17 at 03:31
... FWIW, the only case I know will definitely allow renaming a file but not deleting it is if the file is opened with `FILE_SHARE_DELETE` but has a file mapping associated with it. — Harry Johnston, Apr 24 '17 at 03:33
(Since your use case is files that are only being held open for a short period, have you considered using ordinary file rename operations and retrying as necessary whenever you receive a sharing violation?) — Harry Johnston, Apr 24 '17 at 03:36
@HarryJohnston Why is there a difference between my use case and my sample code? (I did split my sample code into two separate processes just in case it matters, but it made no difference). I have not yet tested `ReplaceFile` in the real code because it's so hard to get clear results (the problem is highly intermittent). I can do it of course, but I was hoping to have a simple test first to make sure it has a hope of working. — max, Apr 24 '17 at 03:36
As for retrying, of course, it will work. But we have a large number of files, so it would potentially cause unnecessary slowdown. That's what we'll do if there's no cleaner solution. — max, Apr 24 '17 at 03:37
If the other process has an opportunistic lock on the file, that will change the behaviour of the API, although I'm not at all sure that would explain the apparent discrepancy. Also, I wouldn't be at all surprised if ReplaceFile automatically retries an operation that fails with a sharing violation. That's just a guess. — Harry Johnston, Apr 24 '17 at 03:42
... IMO, the performance impact of retrying yourself should typically be negligible, but you could further minimize it by queuing up the failed operations rather than waiting for each one to complete individually. — Harry Johnston, Apr 24 '17 at 03:44
`ReplaceFile` fail because it try open *lpReplacedFileName* with `DELETE` access mask. but this give error `ERROR_SHARING_VIOLATION` if file already open without `FILE_SHARE_DELETE` — RbMm, Apr 24 '17 at 07:07
@RbMm but what about the story http://stackoverflow.com/questions/8958094/reliable-file-saving-file-replace-in-a-busy-environment/8964246#8964246 about ReplaceFile, if called with three arguments, allowing to rename files that cannot be deleted? — max, Apr 24 '17 at 08:06
this answer wrong in this part. for rename file need call [ZwSetInformationFile](https://msdn.microsoft.com/en-us/library/windows/hardware/ff567096(v=vs.85).aspx) with *FileRenameInformation* FileInformationClass. for this `The caller must have DELETE access to the file.` from another side are caller can got `DELETE` access determinate. but we fail got `DELETE` access if fail already open with read or write access and without `FILE_SHARE_DELETE` — RbMm, Apr 24 '17 at 08:40

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

I had experienced several times similar issues using Windows DLLs.

For performances optimization, security purposes and a bounch of other low level but nonetheless important details I didn't dig into, DLLs in Windows are loaded once (per platform/version/culture/...). So when you refere them, as you did in your code, if they have been already loaded by some other process they have also already loaded their data, security context and so on.

In this case the call to the DLL even if it's performed by your process has a different context: here is where the sharing thing starts making sense.

As you already know, Python is a general purpose, multiplatform programming language, so it's main aim is to give you an interface to the underlying OS features indipendently of how it exposes them.

Well, it turns out that the windows CreateFile function as a significant high number of options that can be delivered to it and does not apply to other OS, one of which is dwShareMode that dictates how the file is shared with other processes. (Here the ufficial documentation).

If this parameter is zero and CreateFile succeeds, the file or device cannot be shared and cannot be opened again until the handle to the file or device is closed.

You cannot request a sharing mode that conflicts with the access mode that is specified in an existing request that has an open handle. CreateFile would fail and the GetLastError function would return ERROR_SHARING_VIOLATION.

I think core python developers, if using that function at all (not the posix one if supported by windows), had left the value of that variable to 0 leading to the behaviour you're experiencing.

In order for your process to be able to call the DLLs functions that deals with the files created by your own code you have to be sure you do not have "non shared" descriptors open to them.

You have at least two options:

Create the file calling directly the Windows API and specifying the sharing behaviour you desire.
Opening file through context manager

Solution 2 is easier and more "pythonic" so I'll go with it:

import os
import ctypes

fname1 = 'test1.txt'
fname2 = 'test2.txt'

with open(fname1, 'w') as f1:
   f1.write(fname1)

with open(fname2, 'w') as f2:
   f2.write(fname2)

ctypes.windll.kernel32.ReplaceFileW(fname2, fname1, 'tmp123', 0, None, None)

Why does ReplaceFile fail with ERROR_SHARING_VIOLATION?

1 Answers1

Linked