Opening a file on Windows with exclusive locking in Python

Question

I have a question quite similar to this question, where I need the follow conditions to be upheld:

• If a file is opened for reading, that file may only be opened for reading by any other process/program
• If a file is opened for writing, that file may only be opened for reading by any other process/program

The solution posted in the linked question uses a third party library which adds an arbitrary .LOCK file in the same directory as the file in question. It is a solution that only works wrt to the program in which that library is being used and doesn't prevent any other process/program from using the file as they may not be implemented to check for a .LOCK association.

In essence, I wish to replicate this result using only Python's standard library.

BLUF: Need a standard library implementation specific to Windows for exclusive file locking

To give an example of the problem set, assume there is:

• 1 file on a shared network/drive
• 2 users on separate processes/programs

Suppose that User 1 is running Program A on the file and at some point the following is executed:

with open(fp, 'rb') as f:
    while True:
        chunk = f.read(10)
        if chunk:
            # do something with chunk
        else:
            break 

Thus they are iterating through the file 10 bytes at a time.

Now User 2 runs Program B on the same file a moment later:

with open(fp, 'wb') as f:
    for b in data:  # some byte array
        f.write(b)

On Windows, the file in question is immediately truncated and Program A stops iterating (even if it wasn't done) and Program B begins to write to the file. Therefore I need a way to ensure that the file may not be opened in a different mode that would alter its content if previously opened.

I was looking at the msvcrt library, namely the msvcrt.locking() interface. What I have been successful at doing is ensuring that a file opened for reading can be locked for reading, but nobody else can read the file (as I lock the entire file):

>>> f1 = open(fp, 'rb')
>>> f2 = open(fp, 'rb')
>>> msvcrt.locking(f1.fileno(), msvcrt.LK_LOCK, os.stat(fp).st_size)
>>> next(f1)
b"\x00\x05'\n"
>>> next(f2)
PermissionError: [Errno 13] Permission denied

This is an acceptible result, just not the most desired.

In the same scenario, User 1 runs Program A which includes:

with open(fp, 'rb') as f
    msvcrt.locking(f.fileno(), msvcrt.LK_LOCK, os.stat(fp).st_size)
    # repeat while block
    msvcrt.locking(f.fileno(), msvcrt.LK_UNLCK, os.stat(fp).st_size)

Then User 2 runs Program B a moment later, the same result occurs and the file is truncated.

At this point, I would've liked a way to throw an error to User 2 stating the file is opened for reading somewhere else and cannot be written at this time. But if User 3 came along and opened the file for reading, then there would be no problem.

Update:

A potential solution is to change the permissions of a file (with exception catching if the file is already in use):

>>> os.chmod(fp, stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
>>> with open(fp, 'wb') as f:
        # do something
PermissionError: [Errno 13] Permission denied <fp>

This doesn't feel like the best solution (particularly if the users didn't have the permission to even change permissions). Still looking for a proper locking solution but msvcrt doesn't prevent truncating and writing if the file is locked for reading. There still doesn't appear to be a way to generate an exclusive lock with Python's standard library.

Answer 1

For those who are interested in a Windows specific solution:

import os
import ctypes
import msvcrt
import pathlib

# Windows constants for file operations
NULL = 0x00000000
CREATE_ALWAYS = 0x00000002
OPEN_EXISTING = 0x00000003
FILE_SHARE_READ = 0x00000001
FILE_ATTRIBUTE_READONLY = 0x00000001  # strictly for file reading
FILE_ATTRIBUTE_NORMAL = 0x00000080  # strictly for file writing
FILE_FLAG_SEQUENTIAL_SCAN = 0x08000000
GENERIC_READ = 0x80000000
GENERIC_WRITE = 0x40000000

_ACCESS_MASK = os.O_RDONLY | os.O_WRONLY
_ACCESS_MAP = {os.O_RDONLY: GENERIC_READ,
               os.O_WRONLY: GENERIC_WRITE
               }

_CREATE_MASK = os.O_CREAT | os.O_TRUNC
_CREATE_MAP = {NULL: OPEN_EXISTING,
               os.O_CREAT | os.O_TRUNC: CREATE_ALWAYS
               }

win32 = ctypes.WinDLL('kernel32.dll', use_last_error=True)
win32.CreateFileW.restype = ctypes.c_void_p
INVALID_FILE_HANDLE = ctypes.c_void_p(-1).value


def _opener(path: pathlib.Path, flags: int) -> int:

    access_flags = _ACCESS_MAP[flags & _ACCESS_MASK]
    create_flags = _CREATE_MAP[flags & _CREATE_MASK]

    if flags & os.O_WRONLY:
        share_flags = NULL
        attr_flags = FILE_ATTRIBUTE_NORMAL
    else:
        share_flags = FILE_SHARE_READ
        attr_flags = FILE_ATTRIBUTE_READONLY

    attr_flags |= FILE_FLAG_SEQUENTIAL_SCAN

    h = win32.CreateFileW(path, access_flags, share_flags, NULL, create_flags, attr_flags, NULL)

    if h == INVALID_FILE_HANDLE:
        raise ctypes.WinError(ctypes.get_last_error())

    return msvcrt.open_osfhandle(h, flags)


class _FileControlAccessor(pathlib._NormalAccessor):

    open = staticmethod(_opener)


_control_accessor = _FileControlAccessor()


class Path(pathlib.WindowsPath):

    def _init(self) -> None:

        self._closed = False
        self._accessor = _control_accessor

    def _opener(self, name, flags) -> int:

        return self._accessor.open(name, flags)