2

The pros and cons of multiprocessing versus threading are discussed here: Multiprocessing vs Threading Python

I'd like to use multiprocessing to perform a task in the background, while continuing on with the rest of the program. This problem was solved in the following question question using threading: background function in Python 

import threading

def function_A(some_args):
    # some stuff

def function_B(some_args):
    #do some stuff
    my_thread = threading.Thread(target=function_A, args=my_args)
    my_thread.start()
    #continue doing stuff

Here's my attempt using multiprocessing:

import multiprocessing as mp

def function_A(some_args):
    # some stuff

worker_pool = mp.Pool(processes=4)

def function_B(some_args, worker_pool):
    # do some stuff
    worker_pool.apply_async(function_A, args=my_args)
    # continue doing stuff

I tried this out in a test script:

import threading
import multiprocessing
import time


def function_A(task_description, sleep_time):
    time.sleep(sleep_time)
    print "function_A with task {} completed!\n".format(task_description)

def function_threading(sleep_time):
    st = time.time()

    for x in xrange(4):
        print "Calling A and continuing (threading)..."
        print "Time between calls: ", round(time.time() - st, 3)        
        a_thread = threading.Thread(target=function_A, args=("threading {}".format(x), sleep_time,))
        a_thread.start()
        st = time.time()

def function_multiprocessing(sleep_time, worker_pool):
    st = time.time()

    for x in xrange(4):
        print "Calling A and continuing (multiprocessing)..."
        print "Time between calls: ", round(time.time() - st, 3) 
        worker_pool.apply_async(function_A, args=("multiprocessing {}".format(x), sleep_time,))
        st = time.time()

sleep_time = 2.5

# if __name__ == "__main__" statement required for multiprocessing on Windows
if __name__ == "__main__":
    print "================================"
    print "Threading test."
    function_threading(sleep_time)
    print "================================"
    print "Multiprocessing test."
    worker_pool = multiprocessing.Pool(processes=4)
    function_multiprocessing(sleep_time, worker_pool)
    print "================================"

Here's the output I get:

================================
Threading test.
Calling A and continuing (threading)...
Time between calls:  0.0
Calling A and continuing (threading)...
Time between calls:  0.0
Calling A and continuing (threading)...
Time between calls:  0.0
Calling A and continuing (threading)...
Time between calls:  0.001
================================
Multiprocessing test.
Calling A and continuing (multiprocessing)...
Time between calls:  0.0
Calling A and continuing (multiprocessing)...
Time between calls:  0.001
Calling A and continuing (multiprocessing)...
Time between calls:  0.0
Calling A and continuing (multiprocessing)...
Time between calls:  0.0
================================

function_A with task threading 0 completed!
function_A with task threading 2 completed!
function_A with task threading 1 completed!
function_A with task threading 3 completed!

Observations:

  • there are child processes left hanging about even after the main process is killed -- is this from threading, or multiprocessing? I have to CTRL+ALT+DELETE and manually kill them

  • the multiprocessing tasks never print

What needs to be fixed in order for the script to work? Or, put another way, how can I run and manage background processes using multiprocessing?

Community
  • 1
  • 1
bzm3r
  • 3,113
  • 6
  • 34
  • 67
  • What, exactly, do you imagine `function_A()` to be doing? – zxq9 Nov 17 '15 at 22:58
  • But it didn't work. You never wait on the `AsyncResult` object returned by `apply_async` so your parent just exits. – tdelaney Nov 17 '15 at 22:59
  • @zxq9 `function_A` could be making a video (using `subprocess` to call `ffmpeg`) out of images that `function_B` has left behind using data separate from what the main process is current working with – bzm3r Nov 17 '15 at 23:46
  • @tdelaney `function_A` doesn't return any results, so why should I wait for any results from it? – bzm3r Nov 17 '15 at 23:47
  • 3
    The problem is that the child processes are killed when the parent exits. You have an infinite loop in `function_A`. Fix that and add a print statement at the bottom. It won't print anything. Then add a time.sleep of. say, 15 seconds at the end of the main program and you will see the print. – tdelaney Nov 18 '15 at 00:19
  • @tdelaney I totally see what you mean; let me try that out again! – bzm3r Nov 18 '15 at 02:04
  • @tdelaney I fixed the infinite loop problem, and added some relevant print statements (see the edited code) based on your suggestion. As output, I see that the threading calls print something, but the multiprocessing calls never print anything...what's up with that? – bzm3r Nov 18 '15 at 02:20
  • It's been a long time since I did any multi processing in Python, but I seem to remember something about the child process not having a handle to stdout. You could try printing to a log file and see if that works. – visibleman Nov 18 '15 at 02:27
  • This may be irrelevant, but I would like to mention http://www.celeryproject.org – binithb Nov 19 '15 at 17:15

0 Answers0