I'm trying to create a context manager that uses mmap which is itself is a context manager. Initially I had an dumb open file problem Why isn't mmap closing associated file (getting PermissionError: [WinError 32])? and an answer quickly explained why it wasn't working as desired.
Given that information, I've attempted two different ways to correct the issue, but neither one has worked.
The first approach was to use thecontextlib's @contextmanager decorator:
from contextlib import contextmanager
import os
import mmap
#contextmanager
def memory_map(filename, access=mmap.ACCESS_WRITE):
size = os.path.getsize(filename)
fd = os.open(filename, os.O_RDWR)
print('about to yield')
with mmap.mmap(fd, size, access=access) as m:
yield m
print('in finally clause')
os.close(fd) # Close the associated file descriptor.
test_filename = 'data'
# First create the test file.
size = 1000000
with open(test_filename, 'wb') as f:
f.seek(size - 1)
f.write(b'\x00')
# Read and modify mmapped file in-place.
with memory_map(test_filename) as m: # Causes AttributeError: __enter__
print(len(m))
print(m[0:10])
# Reassign a slice.
m[0:11] = b'Hello World'
# Verify that changes were made
print('reading back')
with open(test_filename, 'rb') as f:
print(f.read(11))
# Delete test file.
# Causes:
# PermissionError: [WinError 32] The process cannot access the file because it
# is being used by another process: 'data'
os.remove(test_filename)
But it results in:
Traceback (most recent call last):
File "memory_map.py", line 27, in <module>
with memory_map(test_filename) as m: # Causes AttributeError: __enter__
AttributeError: __enter__
In the next attempt I tried explicitly creating a context manager class:
import os
import mmap
class MemoryMap:
def __init__(self, filename, access=mmap.ACCESS_WRITE):
print('in MemoryMap.__init__')
size = os.path.getsize(filename)
self.fd = os.open(filename, os.O_RDWR)
self.mmap = mmap.mmap(self.fd, size, access=access)
def __enter__(self):
print('in MemoryMap.__enter__')
return self.mmap
def __exit__(self, exc_type, exc_value, traceback):
print('in MemoryMap.__exit__')
os.close(self.fd) # Close the associated file descriptor.
print(' file descriptor closed')
test_filename = 'data'
# First create the test file.
size = 1000000
with open(test_filename, 'wb') as f:
f.seek(size - 1)
f.write(b'\x00')
# Read and modify mmapped file in-place.
with MemoryMap(test_filename) as m:
print(len(m))
print(m[0:10])
# Reassign a slice.
m[0:11] = b'Hello World'
# Verify that changes were made
print('reading back')
with open(test_filename, 'rb') as f:
print(f.read(11))
# Delete test file.
# Causes PermissionError: [WinError 32] The process cannot access the file
# because it is being used by another process: 'data'
os.remove(test_filename)
This makes it further, but the PermissionError is back—which really confuses me because the file descriptor was closed in that version as you can see in the output produced:
in MemoryMap.__init__
in MemoryMap.__enter__
1000000
b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
in MemoryMap.__exit__
file descriptor closed
reading back
b'Hello World'
Traceback (most recent call last):
File "memory_map2.py", line 47, in <module>
os.remove(test_filename)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'data'
So it seems I'm stuck again. Any ideas on what's wrong (as well as how to fix it)? Also, in the event they can both be fixed, which one is better if you have an opinion?
Solutions
There was an error in both snippets. This first was a simple typographical error. The contextmanger decorator was commented out. Should have been:
@contextmanager # Leading "#" changed to "@".
def memory_map(filename, access=mmap.ACCESS_WRITE):
size = os.path.getsize(filename)
fd = os.open(filename, os.O_RDWR)
...
In the second it was because the mmap itself was not being closed in the __exit__() method, just the associated file descriptor. That never occurred to me because the exception raised was the same as in the first case.
def __exit__(self, exc_type, exc_value, traceback):
print('in MemoryMap.__exit__')
self.mmap.close() # ADDED.
os.close(self.fd) # Close the associated file descriptor.
print(' file descriptor closed')
