Python: Opening a file without creating a lock

Question

I'm trying to create a script in Python to back up some files. But, these files could be renamed or deleted at any time. I don't want my script to prevent that by locking the file; the file should be able to still be deleted at any time during the backup.

How can I do this in Python? And, what happens? Do my objects just become null if the stream cannot be read?

Thank you! I'm somewhat new to Python.

The behaviour depends on the operating system. Which OS(es) do you need to support? — NPE, Jan 17 '13 at 21:39
I do not think that you can do much about it. If the files get deleted while you are reading it you'll probably receive some `IOError` or `OSError`(in python there is no such thing as `null`. Even if the file is deleted the file object python is managing is not deleted and when doing the next system (or C)call it will fail thus raising an exception). — Bakuriu, Jan 17 '13 at 21:42
Normal `open()` on Python does not create locks. Further, advisory file system locking does not prevent deletion or renaming of a file, only reading and writing (and even that depends on the application using `flock()` correctly). — Francis Avila, Jan 17 '13 at 21:44
@FrancisAvila that's the situation on Unix, yes. On Windows, open files are "locked" so they can't be deleted (IIRC, there is a sharing mode for `CreateFile` to allow deletion, but it has some obscure problems...) — Anton Kovalenko, Jan 17 '13 at 21:47
@Bakuriu: Reading from a deleted file in Linux works fine if you opened it before it was deleted. — Martijn Pieters, Jan 17 '13 at 21:48
@MartijnPieters That's why I said "probably". What I wanted to point out is that when the file is deleted python's file object still exists. Its behaviour will depend on the OS, but it wont simply become null and crash the whole interpreter. — Bakuriu, Jan 17 '13 at 22:08
I think I need something aside from VSS. @AntonKovalenko Others talk about the sharing flag when opening a file in Windows. Can we do this in Python? In .NET there's a FileShare.Delete flag, and here's the CreateFile link: http://msdn.microsoft.com/en-us/library/windows/desktop/aa363858(v=vs.85).aspx — SilentSteel, Jul 28 '13 at 09:50
Hmm.. I'm getting closer now, looks like I can use win32file.CreateFile to use the Windows specific dwSharingMode flag: http://docs.activestate.com/activepython/2.4/pywin32/win32file__CreateFile_meth.html — SilentSteel, Jul 28 '13 at 09:59

score 11 · Accepted Answer · edited Dec 06 '21 at 13:44

As mentioned by @kindall, this is a Windows-specific issue. Unix OSes allow deleting.

To do this in Windows, I needed to use win32file.CreateFile() to use the Windows-specific dwSharingMode flag (in Python's pywin32, it's just called shareMode).

Rough Example:

import msvcrt
import os
import win32file

py_handle = win32file.CreateFile(
    'filename.txt',
    win32file.GENERIC_READ,
    win32file.FILE_SHARE_DELETE
        | win32file.FILE_SHARE_READ
        | win32file.FILE_SHARE_WRITE,
    None,
    win32file.OPEN_EXISTING,
    win32file.FILE_ATTRIBUTE_NORMAL,
    None
)
try:
    with os.fdopen(
        msvcrt.open_osfhandle(py_handle.handle, os.O_RDONLY)
    ) as file_descriptor:
        ... # read from `file_descriptor`
finally:
    py_handle.Close()

Note: if you need to keep the win32-file open beyond the lifetime of the file-handle object returned, you should invoke PyHandle.detach() on that handle.

Then you can use `os.fdopen(msvcrt.open_osfhandle(file_handle, os.O_RDONLY))` to transform that handle into a standard file object. Also, `os.fdopen()` can be used with `with` too, just as `open()`. — user, Oct 06 '16 at 01:15

kindall · Answer 2 · 2013-01-18T05:13:23.943

5

On UNIX-like OSs, including Linux, this isn't an issue. Well, some other program could write to the file at the same time you're reading it, which could cause problems (the file you are copying could end up corrupted) but this is solvable with a verification pass.

On Windows, use Volume Snapshot Service (aka Volume Shadow Copy). VSS creates a snapshot of the volume at a moment in time, and you can open files on the snapshot without locking the files on the original volume. A quick Google found a Python module for doing copies using VSS here: http://sourceforge.net/projects/pyvss/

edited Jan 18 '13 at 05:13

answered Jan 17 '13 at 22:04

kindall

178,883
35
278
309

Accepted as answer, but I think I need something aside from VSS. – SilentSteel Jul 28 '13 at 09:47

Python: Opening a file without creating a lock

2 Answers2

Linked