-1

I have recently start working with Python multiprocessing module. I understand explanation of queues, but recently I found on https://pymotw.com/2/multiprocessing/communication.html that queues don't need to be pass as args to Proccess constructor method, e.g.

p = Process(target=f, args=(q,)),

instead, it seems that they are globally shared. I thought that this is only the case when we have managed queues, i.e.

queue = manager.Queue()

Can someone help me to understand this?

user3616359
  • 359
  • 3
  • 20
  • 1
    A quick skim of that article doesn't show what you describe. Everywhere I see a process using a queue, it receives that queue as an argument explicitly. – user2357112 Apr 13 '17 at 19:48
  • Not in the part where class Consumer(multiprocessing.Process) is defined. The queue is a member of that class, which is inherited from processes. – user3616359 Apr 13 '17 at 20:24
  • 1
    That `Consumer` class takes all queues it uses as arguments. – user2357112 Apr 13 '17 at 20:40
  • Now I see, what you mean. "The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments" So, all these queues are actually passed as an arguments. – user3616359 Apr 13 '17 at 21:53

1 Answers1

2

In Unix, a child process is created with fork().

In Windows, a child process is created by invoking the same script with special arguments.

In both cases, there may be the q variable in the child process because it inherited the state or because the relevant code has run before execution reached the worker function.

But that is not enough. An IPC needs to be set up between the processes for it to play its role as a communication channel. Otherwise, it's just a regular local object.

When in doubt, see the official documentation which is the authoritative information source and is generally of exceptional quality. With multiprocessing, it's especially important to stick to the docs because due to its quirky nature, various things may seem to work but break in unpredictable ways.

Community
  • 1
  • 1
ivan_pozdeev
  • 33,874
  • 19
  • 107
  • 152
  • Just for clarification, this can work because q is inherited in the child process and IPC is established, but it is not advised to work like this? – user3616359 Apr 13 '17 at 20:29
  • @user3616359 Exactly - even if it happens to work, "it is not advised to work like this". – ivan_pozdeev Apr 13 '17 at 20:47
  • @user3616359 Debugging issues involving multiple interacting processes is hard enough even without as much implicit logic under the hood as in `multiprocessing` because they usually can't be reproduced reliably. So, you really should avoid them from the get-go as much as possible. – ivan_pozdeev Apr 13 '17 at 21:01
  • Thanks :) However, if this particular design is sometimes preferred, this derived class which inherits from Process, what is the best way to pass queue to run method? – user3616359 Apr 13 '17 at 21:39
  • @user3616359 Ask this as another question. Not only SO is one-concern-per-question, you never gave any details about your "design" that you're referring to, so I can't say anything. – ivan_pozdeev Apr 13 '17 at 21:41