Parallel execution of 2 separate Python scripts using subprocess from the main Python script

Question

Using Python, I want to start two subprocesses in parallel. One will start an HTTP server, while the other will start execution of another program (which is a Python script generated by Selenium IDE plugin to open Firefox, navigate to a website, and do some interactions). On the other hand, I want to stop execution of the first subprocess (the HTTP server) when the second subprocess is finished executing.

The logic of my code is that the Selenium script will open a website. The website will automatically make a few GET calls to my HTTP server. After the Selenium script is finished executing, the HTTP server is supposed to be closed so that it can log all the captured requests in a file.

Here is my code:

class Myclass(object):

    HTTPSERVERPROCESS = ""

    def startHTTPServer(self):
        print "********HTTP Server started*********"
        try:
            self.HTTPSERVERPROCESS=subprocess.Popen('python CustomSimpleHTTPServer.py', \
                            shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
            self.HTTPSERVERPROCESS.communicate()
        except Exception as e:
            print "Exception captured while starting HTTP Server process: %s\n" % e

    def startNavigatingFromBrowser(self):
        print "********Opening firefox to start navigation*********"
        try:
            process=subprocess.Popen('python navigationScript.py', \
                            shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
            process.communicate()
            process.wait()
        except Exception as e:
            print "Exception captured starting Browser Navigation process : %s\n" % e
        try:
            if process.returncode==0:
                print "HTTPSERVEPROCESS value: %s" % self.HTTPSERVERPROCESS.returncode
                print self.HTTPSERVERPROCESS
                self.HTTPSERVERPROCESS.kill()
                #print "HTTPSERVEPROCESS value: %s" % self.HTTPSERVERPROCESS.returncode
        except Exception as e:
            print "Exception captured while killing HTTP Server process : %s\n" % e

    def startCapture(self):
        print "********Starting Parallel execution of Server initiation and firefox navigation script*********"
        t1 = threading.Thread(target=self.startHTTPServer())
        t2 = threading.Thread(target=self.startNavigatingFromBrowser())
        t1.start()
        t2.start()
        t2.join()

Note: Execution starts by calling startCapture()

The problem is that I get the following in my terminal upon running the above code:

********Starting Parallel execution of HTTP Server initiation and firefox navigation script*********
********HTTP Server started*********
********Opening firefox to start navigation*********


Process finished with exit code 0

My program finishes execution even when the thread started for startNavigatingFromBrowser() is still active. I can see that Firefox is navigating through the website even after I get "Process finished with exit code 0" for my program. Due to this I am not able to detect when my browser navigation thread finishes execution. (This is necessary because I am using process.returncode returned from the navigationScript subprocess to kill my HTTP server process).

What changes should I make to the code so that I can successfully detect when the Selenium navigation subprocess is finished executing so that I can stop my HTTP server?

Any luck now? I replicated the structure with the wait() call, and it works for me. — Max Uppenkamp, Jul 27 '16 at 12:14

Max Uppenkamp · Accepted Answer · 2016-07-27T11:24:54.630

1

Call t2.join() before exiting your program. This waits for the navigation thread to terminate, before execution continues.

Edit:

Your navigation thread terminates immediately because you are not waiting for the child process to exit. This should solve the issue:

process=subprocess.Popen('python navigationScript.py', \
                        shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
process.wait()

This will halt the thread until the subprocess is done.

edited Jul 27 '16 at 11:24

answered Jul 27 '16 at 11:15

Max Uppenkamp

974
4
16

Have updated my code above with your suggestion. But even after adding t2.join(), the result is the same. – Mr. Bing Jul 27 '16 at 11:20
Have updated my code with your suggestion. But I am stuck with another problem now. I get this exception: "Exception captured while killing HTTP Server process : [Errno 3] No such process" when execution point reaches self.HTTPSERVERPROCESS.kill() – Mr. Bing Aug 02 '16 at 16:06
have created a new thread for this problem here: http://stackoverflow.com/questions/38726804/unable-to-kill-python-subprocess-using-process-kill-or-process-terminate-or – Mr. Bing Aug 02 '16 at 17:51

Parallel execution of 2 separate Python scripts using subprocess from the main Python script

1 Answers1