Does a queue make sense for multiprocessing even when you only expect a single result back?

Question

I want to execute a function in another process and get a single result back (either true or false). I know the common way of getting results back from multiprocessing is using a queue, but does it make sense if I only expect a single result back?

p = Process(target=my_function_that_returns_boolean, args=(self, args))
p.start()
p.join()
# success = p.somehow_get_the_result_back

score 2 · Answer 1 · answered Jun 18 '14 at 20:06

If you prefer to use Process rather than Pool, the documentation tells us that there are two ways to exchange objects between processes.

The first is Queue which you have already seen.

The second is Pipe, which the documentation provides an example for. I have slightly modified the example to show your case of returning a boolean.

from multiprocessing import Process, Pipe


def Foo(conn):
  # Do necessary processing here
  # ....
  # Instead of Return True, we send true
  #return True
  conn.send(True)


parent_conn, child_conn = Pipe()
p = Process(target=Foo, args=(child_conn,))
p.start()
print parent_conn.recv()
p.join()

`Pipe` is faster than `Queue`, for what it's worth. It won't make any noticeable difference if you're only returning a single object, though. — dano, Jun 19 '14 at 15:12

score 1 · Answer 2 · answered Jun 18 '14 at 06:05

1

Queues are used to synchronize access to shared resources in a parallel environment. Common scenarios are when many workers consume tasks from a shared pool or when one execution line creates tasks and another consumes them.

If I understand correctly, it isn't an issue here. So there is no need to use queues. The only synchronization mechanism you need is one that tells one process that the other is done. This is achieved by using join().

Unless there is a real problem just keep things as simple as possible.

answered Jun 18 '14 at 06:05

Xyand

4,470
4
36
63

But join doesn't return anything, how do I get the result back? – Clash Jun 18 '14 at 06:15
@merlin2011, I only see `result_queue.get()`, I must be blind. Where is the object with a `get()` method? – Clash Jun 18 '14 at 15:34
1

@Clash You're not missing anything, there is no `get` to call when you use a `Process`. You should use a `Queue` or `Pipe` to get the result back. – dano Jun 19 '14 at 15:11

score 0 · Answer 3 · answered Jun 18 '14 at 06:55

You can use a Pool which returns an AsyncResult object

from multiprocessing import Pool

pool = Pool(processes=1)              
result = pool.apply_async(my_function_that_returns_boolean, [(self, args),])    
success = result.get(timeout=None)

Does a queue make sense for multiprocessing even when you only expect a single result back?

3 Answers3