Python multiprocessing/threading blocking main thread

Question

I’m trying to write a program in Python. What I want to write is a script which immediately returns a friendly message to the user, but spawns a long subprocess in the background that takes with several different files and writes them to a granddaddy file. I’ve done several tutorials on threading and processing, but what I’m running into is that no matter what I try, the program waits and waits until the subprocess is done before it displays the aforementioned friendly message to the user. Here’s what I’ve tried:

Threading example:

#!/usr/local/bin/python
import cgi, cgitb
import time
import threading

class TestThread(threading.Thread):
     def __init__(self):
         super(TestThread, self).__init__()

     def run(self):
        time.sleep(5)
        fileHand = open('../Documents/writable/output.txt', 'w')
        fileHand.write('Big String Goes Here.')
        fileHand.close()

print 'Starting Program'

thread1 = TestThread()
#thread1.daemon = True
thread1.start()

I’ve read these SO posts on multithreading How to use threading in Python? running multiple threads in python, simultaneously - is it possible? How do threads work in Python, and what are common Python-threading specific pitfalls?

The last of these says that running threads concurrently in Python is actually not possible. Fair enough. Most of those posts also mention the multiprocessing module, so I’ve read up on that, and it seems fairly straightforward. Here’s the some of the resources I’ve found:

How to run two functions simultaneously Python Multiprocessing Documentation Example https://docs.python.org/2/library/multiprocessing.html

So here’s the same example translated to multiprocessing:

#!/usr/local/bin/python
import time
from multiprocessing import Process, Pipe

def f():
    time.sleep(5)
    fileHand = open('../Documents/writable/output.txt', 'w')
    fileHand.write('Big String Goes Here.')
    fileHand.close()

if __name__ == '__main__':
    print 'Starting Program'
    p = Process(target=f)
    p.start()

What I want is for these programs to immediately print ‘Starting Program’ (in the web-browser) and then a few seconds later a text file shows up in a directory to which I’ve given write privileges. However, what actually happens is that they’re both unresponsive for 5 seconds and then they print ‘Starting Program’ and create the text file at the same time. I know that my goal is possible because I’ve done it in PHP, using this trick:

//PHP
exec("php child_script.php > /dev/null &");

And I figured it would be possible in Python. Please let me know if I’m missing something obvious or if I’m thinking about this in the completely wrong way. Thanks for your time!

(System information: Python 2.7.6, Mac OSX Mavericks. Python installed with homebrew. My Python scripts are running as CGI executables in Apache 2.2.26)

You may be running into an issue with the GIL. I've overcome this in my own program by inheriting from the `multiprocessing.Process` class rather than `threading.Thread` and that works. I don't call `p = Process(target=f)`. — Aleksander Lidtke, Apr 11 '14 at 15:44
also the fact that it shows "Starting program" after a while could be because of stdout buffering. try using sys.stdout.flush() — fabrizioM, Apr 11 '14 at 15:46

score 0 · Answer 1 · answered Apr 26 '14 at 15:52

Ok- I think I found the answer. Part of it was my own misunderstanding. A python script can't simply return message to a client-side (ajax) program but still be executing a big process. The very act of responding to the client means that the program has finished, threads and all. The solution, then, is to use the python version of this PHP trick:

//PHP
exec("php child_script.php > /dev/null &");

And in Python:

#Python
subprocess.call(" python worker.py > /dev/null &", shell=True)

It starts an entirely new process outside the current one, and it will continue after the current one has ended. I'm going to stick with Python because at least we're using a civilized api function to start the worker script instead of the exec function, which always made me uncomfortable.

Python multiprocessing/threading blocking main thread

1 Answers1