0

While experimenting with some of the code from the Reading Binary Data into a Mutable Buffer section of the O'Reilly website, I added a line at the end to remove the test file that was created.

However this always results in the following error:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'data'

I don't understand this behavior because the with memory_map(test_filename) as m: should implicitly close the associated file, but apparently doesn't. I can work around this by saving the file descriptor returned from os.open() and then explicitly closing it with an os.close(fd) after the block of statements in the with suite finishes.

Is this a bug or have I missed something?

Code (with the couple of commented-out lines showing my hacky workaround):

import os
import mmap

test_filename = 'data'

def memory_map(filename, access=mmap.ACCESS_WRITE):
#    global fd  # Save to allow closing.
    size = os.path.getsize(filename)
    fd = os.open(filename, os.O_RDWR)
    return mmap.mmap(fd, size, access=access)

# Create test file.
size = 1000000
with open(test_filename, 'wb') as f:
     f.seek(size - 1)
     f.write(b'\x00')

# Read and modify mmapped file in-place.
with memory_map(test_filename) as m:
    print(len(m))
    print(m[0:10])
    # Reassign a slice.
    m[0:11] = b'Hello World'

# os.close(fd)  # Explicitly close the file descriptor -- WHY?

# Verify that changes were made
print('reading back')
with open(test_filename, 'rb') as f:
     print(f.read(11))

# Delete test file.
# Causes PermissionError: [WinError 32] The process cannot access the file
# because it is being used by another process: 'data'
os.remove(test_filename)
martineau
  • 119,623
  • 25
  • 170
  • 301

1 Answers1

2

From the documentation:

close()

Closes the mmap. Subsequent calls to other methods of the object will result in a ValueError exception being raised. This will not close the open file.

The memory mapping is independent of the file handle. You can use the file handle as a normal file futher on.

Community
  • 1
  • 1
Daniel
  • 42,087
  • 4
  • 55
  • 81
  • OK, thanks for clarifying this issue. Seems kind of obvious now in hindsight. `^_^` – martineau Nov 02 '17 at 20:36
  • mmap duplicates the File handle, which it keeps open for use with its `size` and `resize` methods. It's sometimes frustrating that mmap keeps the duplicated File handle open. Windows would allow deleting the underlying file for a read-only (not writable) mapped view -- but mmap couples closing the File handle to unmapping the view. – Eryk Sun Nov 02 '17 at 22:22
  • @eryksun: the duplicated file handle is handled internally in mmap correctly. I doubt one can delete a mapped file, since the contents is mapped by demand. – Daniel Nov 02 '17 at 22:54
  • Deleting a file that's mapped read-only is possible, but it requires a delete-on-close open (or equivalently `NtDeleteFile`), which works because Windows filesystems only call `MmFlushImageSection` with an `MmFlushForWrite` check in this case instead of the full `MmFlushForDelete` check that gets used by `DeleteFile` (i.e. native `NtSetInformationFile`, `FileDispositionInformation`). In recent versions of Windows this is how cmd.exe deletes files, which is how I discovered this. – Eryk Sun Nov 02 '17 at 23:07
  • Since `resize` doesn't work for a read-only mapping, the only reason to keep the File handle open is for the `size` method, but knowing the underlying file size (not the mapped size) is fairly useless for a read-only mapping. So I think it might have been better to close the File handle for a read-only view and raise an exception for `size()` in this case just like what's already done for `resize` -- or at least provide an option to close the File handle. – Eryk Sun Nov 02 '17 at 23:20