How do you share a variable between two processes in Python?

Question

I have two processes, one adds jobs to a queue and the other takes them off the same queue and runs them. This should work as expected and I'm not sure why the worker never gets any jobs. Here's my code:

from multiprocessing import Process
from Queue import Queue
import time

q = Queue()

def queuer():
    while True:
        q.put("JOB")
        print "Adding JOB"
        time.sleep(1)

def worker():  
    while True:
        if not q.empty():
            item = q.get()
            print "Running", item
        else:
            print "No jobs"
            time.sleep(1)

a = Process(target=queuer)
a.start()

b = Process(target=worker)
b.start()

each process has its own copy of the queue. There is another stackoverflow thread that discusses this: http://stackoverflow.com/questions/11109776/changing-global-variable-when-multiprocessing-in-python — wizard23, Mar 12 '14 at 23:46

Felix · Accepted Answer · 2014-03-12T23:52:38.410

Two things:

You need to pass the Queue as an argument to both processes.
You should use multiprocessing.Queue, not Queue.Queue (which are for Threads)

This code works for me:

from multiprocessing import Process, Queue
import time

def queuer(q):
    while True:
        q.put("JOB")
        print "Adding JOB"
        time.sleep(1)

def worker(q):  
    while True:
        if not q.empty():
            item = q.get()
            print "Running", item
        else:
            print "No jobs"
            time.sleep(1)



if __name__ == '__main__':
    q = Queue()
    a = Process(target=queuer, args=(q,))
    b = Process(target=worker, args=(q,))
    a.start()
    b.start()

score 1 · Answer 2 · answered Mar 13 '14 at 00:00

One possibility is to use the Queue object from the multiprocessing namespace. It's described here: http://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes

So I adapted your code. I only made 2 changes: - Use multiprocessing Queue - Avoid globals and pass the queue as a parameter to the worker and queuer (this is not needed but it's good practice to keep everything tidy)

# use the Queue from the multiprocessing namespace!
from multiprocessing import Process, Queue
import time

q = Queue()

def queuer(q):
    while True:
        q.put("JOB")
        print "Adding JOB"
        time.sleep(1)

def worker(q):  
    while True:
        if not q.empty():
            item = q.get()
            print "Running", item
        else:
            print "No jobs"
            time.sleep(1)

a = Process(target=queuer, args =(q,))
a.start()

b = Process(target=worker, args = (q,))
b.start()

How do you share a variable between two processes in Python?

2 Answers2

Linked