9

I am attempting to pass a shared secret to child processes. In a Linux environment this works. In a Windows environment the child does not receive the shared secret. The three files below are a simple example of what I'm trying to do:

main.py

import multiprocessing
import module1
import module2

if __name__ == "__main__":
    module1.init()
    process = multiprocessing.Process(target=module2.start)
    process.start()
    process.join()

module1.py

import ctypes
import multiprocessing

x = None

def init():
    global x
    x = multiprocessing.Value(ctypes.c_wchar_p, "asdf")

module2.py

import module1

def start():
    print(module1.x.value)

In an Ubuntu 14.04 environment, on Python 3.5, I receive the following output:

$ python3 main.py
asdf

In a CentOS 7 environment, I receiving the following output:

$ python3 main.py
asdf

Using the Windows Subsystem for Linux on Windows 10 (both before and after the Creator Update, so Ubuntu 14.04 and 16.04) I get the following output:

$ python3 main.py
asdf

However, in both Windows 7 and Windows 10 environments, using either 3.5 or 3.6, I am getting an AttributeError instead of the above output:

Process Process-1:
Traceback (most recent call last):
  File "C:\Python\Python35\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\Python\Python35\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "H:\Development\replicate-bug\module2.py", line 5, in start
    print(module1.x.value)
AttributeError: 'NoneType' object has no attribute 'value'

I am using a shared ctype. This value should be inherited by the child process.

Why do I receive this AttributeError in a Windows environment, but not a Linux environment?

Andy
  • 49,085
  • 60
  • 166
  • 233

1 Answers1

4

As mentioned in one of the posts automatically linked on the sidebar, windows does not have the fork systemcall present on *NIX systems.

This implies that instead of sharing global state (like NIX Processes can do), a Windows child process is basically completely separate. This includes modules.

What I suspect is happening is that the module gets loaded anew and the module1 you access inside module2.start isn't quite the module you expected.

The multiprocessing guidelines explicitly mention that module-level constants are exempt from the rule: "variables may not contain what you expect". Well in either case, the solution is to explicitly pass the module you want to the child process like so:

module 2

def start(mod1):
    print(mod1.x.value)

main.py

if __name__ == '__main__':
    module1.init()
    process = multiprocessing.Process(target=module2.start, args=(module1,))
    process.start()
    process.join()
Vogel612
  • 5,620
  • 5
  • 48
  • 73
  • Actually, the solution would be to declare `x = multiprocessing.Value(ctypes.c_wchar_p, "asdf")` in `main.py` and pass it as an argument to the child process (completely doing away with `module1` unless there's a need for functionality not shown in the example). –  May 01 '17 at 16:36
  • @Mego there is such a need – ArtOfCode May 01 '17 at 16:39
  • @ArtOfCode In that case, pass it to `module1` also via `module1.init()`. Then, both processes will have a proper reference to the shared object. –  May 01 '17 at 16:41
  • Does this actually just copy the pointer to the shared memory or does it pickle the entire object it points to (which is how `spawn` normally works) – a spaghetto May 01 '17 at 19:16
  • @quartata the documentation doesn't go into semantics ... [run pydoc](https://docs.python.org/3.5/library/multiprocessing.html#multiprocessing.Process.run) only says the target is invoked with the given args and kwargs. It's not clear on whether those are pickled or passed by reference – Vogel612 May 01 '17 at 19:24
  • I'm trying to understand [the code here](https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Lib/multiprocessing/sharedctypes.py) and it does appear that it's doing something special for pickling but I'm not entirely certain what's going on. – a spaghetto May 01 '17 at 19:45
  • @quartata I believe it's both, actually. If my understanding of the source is correct, the object is pickled, and the pickled representation is stored in shared memory. –  May 02 '17 at 06:04
  • @Mego, the ctypes object's buffer is allocated from shared memory, and the parameters to access it are pickled and piped to the child. That's fine. But `c_char_p` is a pointer (address) referencing the "asdf" string in the parent process, which on Windows will be meaningless in the child and will likely raise an access violation (it did for me on the first try). The OP needs to share an array instead, e.g. `multiprocessing.Array(ctypes.c_char, b'asdf')`. – Eryk Sun May 02 '17 at 08:32