Wait for child using os.system

Question

I use a lot of os.system calls to create background processes inside a for loop. How can I wait for all the background processes to end ?

os.wait tells me there are no child process.

ps: I am using Solaris

here is my code :

#!/usr/bin/python
import subprocess
import os

pids = []
NB_PROC=30

for i in xrange(NB_PROC):
        p = subprocess.Popen("(time wget http://site.com/test.php 2>&1 | grep real )&", shell=True)
        pids.insert(0,p)
        p = subprocess.Popen("(time wget http://site.com/test.php 2>&1 | grep real )&", shell=True)
        pids.insert(0,p)

for i in xrange(NB_PROC*2):
        pids[i].wait()
os.system("rm test.php*")

os.system is not deprecated it seems : http://docs.python.org/library/os.html but even by using subprocess i have the same error — maazza, Aug 22 '12 at 09:47

Roland Smith · Accepted Answer · 2012-08-22T13:12:34.247

6

Normally, os.system() returns when the child process is finished. So there is indeed nothing for os.wait() to do. It is equivalent to subprocess.call().

Use subprocess.Popen() to create background processes, and then the wait() or poll() methods of the Popen objects to wait for them to quit.

By default, Popen does not spawn a shell but executes the program directly. This saves resources and prevents possible shell injection attacks.

According to the documentation for os.system():

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function

If you want to do multiple jobs in parallel, consider using multiprocessing, especially the Pool object. It takes care of a lot of the details of farming work out over several processes.

Edit: Timing the execution of a program;

import time
import subprocess

t1 = time.clock()
t2 = time.clock()
overhead = t2-t1

t1 = time.clock()
subprocess.call(['wget', 'http://site.com/test.php'])
t2 = time.clock()
print 'elapsed time: {:.3f} seconds.'.format(t2-t1-overhead)

edited Aug 22 '12 at 13:12

answered Aug 22 '12 at 10:21

Roland Smith

42,427
3
64
94

@maazza The Python script with `os.system` waits for the shell to terminate. Since the shell command ends in `&`, the shell terminates as soon as it's launched the pipeline, without waiting for it to terminate. – Gilles 'SO- stop being evil' Aug 22 '12 at 11:39
true ! but what about subprocess ? do you think it has the same problem ? – maazza Aug 22 '12 at 12:54
@maazza That's not a problem, it's just doing what you told it to. What are you tring to accomplish? If you want to time the execution of a command, just use `subprocess.call()` in between two `time.clock()` calls. – Roland Smith Aug 22 '12 at 13:07
no it needs to be concurrent requests to a web server , time.clock does not help – maazza Aug 22 '12 at 13:16
1

So what are you trying to do? Stress-test a web-server? In that case I suggest you look at the requests module (http://docs.python-requests.org/en/latest/index.html). To parallelize that, use e.g. the `map()` method from `multiprocessing.Pool`. With that, you can even test several URLs, not just one. – Roland Smith Aug 22 '12 at 14:18

score 3 · Answer 2 · answered Aug 26 '12 at 16:16

the solution was indeed in the subprocess module

#!/usr/bin/python
import subprocess
import os

pids = []
NB_PROC=4
cmd="(time wget http://site.com/test.php 2>&1 | grep elapsed | cut -d ' ' -f 3)"

for i in xrange(NB_PROC):
    p = subprocess.Popen(cmd,stdin=None,stdout=None, shell=True)
    pids.insert(0,p)
    print "request %d processed" % (i+1)


for i in xrange(NB_PROC):
    pids[i].wait()
os.system("rm test.php*")

switched to debian in the process but for some reason sometimes the scripts hangs while sometimes it just runs fine

Wait for child using os.system

2 Answers2

Linked