7

I wanted to implement a simple python program using parallel execution. It's I/O bound, so I figured threads would be appropriate (as opposed to processes). After reading the documentation for Queue and fork, I thought something like the following might work.

q = Queue.Queue()

if os.fork():            # child
    while True:
        print q.get()
else:                    # parent
    [q.put(x) for x in range(10)]

However, the get() call never returns. I thought it would return once the other thread executes a put() call. Using the threading module, things behave more like I expected:

q = Queue.Queue()

def consume(q):
    while True:
        print q.get()

worker = threading.Thread (target=consume, args=(q,))
worker.start()

[q.put(x) for x in range(10)]

I just don't understand why the fork approach doesn't do the same thing. What am I missing?

Brian Hawkins
  • 2,813
  • 2
  • 25
  • 17

3 Answers3

8

The POSIX fork system call creates a new process, rather than a new thread inside the same adress space:

The fork() function shall create a new process. The new process (child process) shall be an exact copy of the calling process (parent process) except as detailed below: [...]

So the Queue is duplicated in your first example, rather than shared between the parent and child.

You can use multiprocessing.Queue instead or just use threads like in your second example :)

By the way, using list comprehensions just for side effects isn't good practice for several reasons. You should use a for loop instead:

for x in range(10): q.put(x)
Community
  • 1
  • 1
Niklas B.
  • 92,950
  • 18
  • 194
  • 224
1

To share the data between unrelated processes, you can use named pipes. Through the os.open() funcion.. http://docs.python.org/2/library/os.html#os.open. You can simply name a pipe as named_pipe='my_pipe' and in a different python programs use os.open(named_pipe, ), where mode is WRONLY and so on. After that you'll make a FIFO to write into the pipe. Don't forget to close the pipe and catch exceptions..

user3238650
  • 11
  • 1
0

Fork creates a new process. The child and parent processes do not share the same Queue: that's why the elements put by the parent process cannot be retrieved by the child.

Christophe Vu-Brugier
  • 2,645
  • 26
  • 21