Python writing into a mapped file - strange behaviour

Question

I have a Python script that can successfully write binary data to a file:

iterable_array = [i + 32 for i in range(50)]
file_handle = open("a.bin", "wb")
bytes_to_write = bytearray(iterable_array)
file_handle.write(bytes_to_write)
file_handle.close()

However, I get the following error:

Traceback (most recent call last):
File "python_100_chars.py", line 20, in <module>
file_handle = open("a.bin", "wb")
OSError: [Errno 22] Invalid argument: 'a.bin'

when I try to write while executing the following program (source originally from Microsoft docs) that creates a file mapping and reads the data after a keypress:

HANDLE hFile = CreateFileA( "a.bin",
                            GENERIC_READ | GENERIC_WRITE,
                            FILE_SHARE_WRITE | FILE_SHARE_READ,
                            NULL,
                            CREATE_ALWAYS,
                            FILE_ATTRIBUTE_NORMAL,
                            NULL);
HANDLE hMapFile;
LPCTSTR pBuf;
hMapFile = CreateFileMapping(
             hFile,    // use paging file
             NULL,                    // default security
             PAGE_EXECUTE_READWRITE,          // read/write access
             0,                       // maximum object size (high-order DWORD)
             BUF_SIZE,                // maximum object size (low-order DWORD)
             szName);                 // name of mapping object

if (hMapFile == NULL)
{
   _tprintf(TEXT("Could not create file mapping object (%d).\n"),
          GetLastError());
   return 1;
}
pBuf = (LPTSTR) MapViewOfFile(hMapFile,   // handle to map object
                     FILE_MAP_ALL_ACCESS, // read/write permission
                     0,
                     0,
                     BUF_SIZE);
if (pBuf == NULL)
{
   _tprintf(TEXT("Could not map view of file (%d).\n"),
          GetLastError());

    CloseHandle(hMapFile);

   return 1;
}


 _getch(); 

 printf("string inside file:%s",(char *)((void *)pBuf));
UnmapViewOfFile(pBuf);

CloseHandle(hMapFile);

I've already tested that I can write into the memory-mapped file (and see the results) with basic I/O in the following way:

HANDLE hFile =  CreateFileA(         "a.bin",
                               GENERIC_WRITE,
          FILE_SHARE_WRITE | FILE_SHARE_READ,
                                        NULL,
                               OPEN_EXISTING,
                      FILE_ATTRIBUTE_NORMAL ,
                                        NULL);
char *p = "bonjour";
DWORD bw;
WriteFile(   hFile,
                 p,
                 8,
               &bw,
               NULL);

What is the python script doing that prevents it from writing?

Thank you for any feedback!

Note that `wb` would *truncate* the file. Perhaps you want to do `r+b`! — Antti Haapala -- Слава Україні, Dec 01 '19 at 14:24
I am not sure what Python is doing, but I believe `w` could translate to `CREATE_ALWAYS` — Antti Haapala -- Слава Україні, Dec 01 '19 at 14:30
Changing to `r+b` fixed it, thank you! Yeah, it is probably setting CREATE_ALWAYS. I'll try to set it in the C program that writes and see if it throws any errors. — thehorseisbrown, Dec 01 '19 at 14:36
Ok, as one would expect: ERROR_USER_MAPPED_FILE (1224) The requested operation cannot be performed on a file with a user-mapped section open. If you make it into an answer I'll accept it btw:) — thehorseisbrown, Dec 01 '19 at 14:41
Answered. I also added a POSIX example and explanation that the python program would run correctly in Linux, OS X etc. — Antti Haapala -- Слава Україні, Dec 01 '19 at 15:15

Antti Haapala -- Слава Україні · Accepted Answer · 2019-12-01T16:14:12.757

I do not have Windows so I cannot quite test the behaviour, but I believe this happens because Windows does not quite follow the POSIX semantics and open(name, 'wb') will, instead of truncating the existing file, open it with CREATE_ALWAYS which would conflict with the file being mapped in another process. The ab mode could work too, but... as the Python documentation says

'a' for appending (which on some Unix systems, means that all writes append to the end of the file regardless of the current seek position).

Unfortunately the open function uses C semantics for the flag, and hence it is not possible to specify the desired mode as "open the file for random access writing only without truncating the file", so the best you can do is "open it for both reading and writing without truncating the file", which would be r+b.

As user Eryk Sun pointed out, the problem is that Windows does not support file truncation at all if there are any existing memory mappings:

If CreateFileMapping is called to create a file mapping object for hFile, UnmapViewOfFile must be called first to unmap all views and call CloseHandle to close the file mapping object before you can call SetEndOfFile.

Likewise there is no mapping of space larger than the existing file - if the mapping is longer than the file size, the file is extended instead...

On POSIXly correct platforms, a program like

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <time.h>
#include <unistd.h>


int main(int argc, char const *argv[])
{
    struct stat s;
    int fd = open("a.bin", O_RDWR);
    fstat(fd, &s);
    unsigned char *d = mmap(0, s.st_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
    getchar();

    for (int i = 0; i < s.st_size; i++) {
        putchar(d[i]);
    }

    putchar('\n');
    fflush(stdout);
    getchar();
}

will not interfere with running the said Python program. However if this C program accesses the file within the within the window while it has been truncated to 0, SIGBUS signal will be raised in the C process.

WINAPI `CREATE_ALWAYS` corresponds to NT [`FILE_OVERWRITE_IF`](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-ntcreatefile), which either truncates an existing file or creates a new file -- in contrast to WINAPI `CREATE_NEW` or NT `FILE_CREATE`, which always creates a new file and fails if the file already exists. The problem is that Windows (WINAPI in particular) does not allow the size of a mapped file to change, so truncating the file is not allowed (e.g. see [`SetEndOfFile`](https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-setendoffile)). — Eryk Sun, Dec 01 '19 at 15:35
@ErykSun Oops the critical word *size* was missing... i.e. the file cannot be even extended? — Antti Haapala -- Слава Україні, Dec 01 '19 at 16:01
*If CreateFileMapping is called to create a file mapping object for hFile, UnmapViewOfFile must be called first to unmap all views and call CloseHandle to close the file mapping object before you can call SetEndOfFile.* — Antti Haapala -- Слава Україні, Dec 01 '19 at 16:03
I recalled that not quite correctly. The file size can be extended, but it cannot shrink to less than the mapped size. So truncation is disallowed, but extension is allowed. I tested and confirm that the mapped file can grow beyond the mapped size via the Python file methods `write` and `truncate` (with a larger size), but attempting to decrease the size via `truncate` fails with a permission error. — Eryk Sun, Dec 01 '19 at 16:09

Python writing into a mapped file - strange behaviour

1 Answers1