python multiprocessing and mmap

Question

My goal is to spawn new process, running in background and pass there mmap object as an argument. Toy example:

from multiprocessing import Process
import mmap
import time 


def modify(buf):
    time.sleep(10)
    buf[0:100] = b"xy" * 50


def main_job(t):
    time.sleep(t)
    
    
if __name__ == "__main__":
    BUF = mmap.mmap(-1, length=100, access=mmap.ACCESS_WRITE)
    BUF[0:100] = b"a" * 100
    p = Process(target=modify, args=(BUF,))
    p.daemon=True
    p.start() #background job started
    main_job(100) #man job started in parallel
    print(BUF[0:100])

TypeError: cannot pickle 'mmap.mmap' object

Here I have main job, which just sleeps and I'd like to spawn process with background job and pass mmap obj as argument to child process. Problem is that mmap object is not pickable and there is no way to use mmap and multiprocessing together. But I have no idea how to create background job where I can pass function and arguments via os.fork() for example. How to solve this problem?

May be of interest to you : [What's the equivalence of os.fork() on Windows with Python?](https://stackoverflow.com/a/52476456/11384184) — Lenormju, Jun 18 '21 at 15:10

Lenormju · Answer 1 · 2021-06-18T15:06:11.850

~~mmap.mmap, in either Windows or Posix version, expects a valid file descriptor number as its first argument. -1 is not a valid file descriptor number.~~
EDIT : -1 is a correct fd value to indicate an anonymous mmap

And I think you did not understand what mmap exactly does. You do not give a mmap object to your other process, instead your main rpocess and your sub process each have to create their own mmap, but both being backed by the same file.

Here is your code with the two fixes (and reduced sleep times) :

from multiprocessing import Process
import mmap
import time


def modify(filename):
    time.sleep(5)
    with open(filename, "r+b") as file, \
            mmap.mmap(file.fileno(), length=100, access=mmap.ACCESS_WRITE) as buf:
        buf[0:100] = b"xy" * 50


def main_job(t):
    time.sleep(t)


if __name__ == "__main__":
    filename = "shared_mem_file"
    with open(filename, "r+b") as file, \
            mmap.mmap(file.fileno(), length=100, access=mmap.ACCESS_WRITE) as BUF:
        BUF[0:100] = b"a" * 100
        BUF.flush()
        p = Process(target=modify, args=(filename,))
        p.daemon = True
        p.start()  # background job started
        main_job(10)  # man job started in parallel
        print(BUF[0:100])
        BUF.close()

output is :

b'xyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxyxy'

Changed in version 3.7: Added ACCESS_DEFAULT constant. To map anonymous memory, -1 should be passed as the fileno along with the length. — Slavka, Jun 18 '21 at 14:22
@Slavka indeed, `-1` is a correct value to indicate an **anonymous mmap**, my bad. I will edit my answer. But anonymous mappings are harder to share : either on Windows you specify a `tagname` (which acts *like* a filename), or you are stuck to use your shared anonymous untagged memory inside the same process, only being able to transmit it unto `fork`ed threads. — Lenormju, Jun 18 '21 at 15:04

python multiprocessing and mmap

1 Answers1