Proper use of os.wait()?

Question

I am trying to solve an issue with automating a series of scripts used in my workplace. I am a beginner so I apologise for what most likely will be an easy question (hopefully), I have read the literature but it didn't quite make sense to me.

Essentially I have a bash script that runs a python script and an R script that needs to be run in order, currently running the code the R script begins before the python is finished and I have been told here than I cannot use the shell wait function as my python script launches child processes and shell wait cannot be used to wait on grandchild processes.

Thats fine, so the solution offered was to make the python and R script wait on their own child processes so that when they exit, the the bash script can properly run in order. Unfortunately I cannot figure out the proper nomenclature of this in my python script.

Here's what i have:

cmd = "python %s/create_keyfile.py %s %s %s %s" %(input, input, input, 
input, input)    
print cmd  
os.system(cmd)

cmd = "python %s/uneak_name_plus_barcode_v2.py %s %s %s %s" %(input, 
input, input, input, input)  
print cmd  
os.system(cmd)

cmd = "python %s/run_production_mode.py  %s %s %s %s %s" %(input, input, 
input, input, input, input)  
print cmd  
os.system(cmd)

Where 'input' is actual inputs in my code, I probably just cant share exactly what we are doing :)

So essentially I am trying to figure out the best way of having the whole script wait on these three scripts before exiting.

os.system() is a blocking call already, meaning it already won't return until completion. Unless your python scripts daemonize, the parent should block on the child before exiting as well. If they do daemonize, you'll have to check for the processes with ps or top. I would speculate that the problem is actually in the BASH script which is probably running your scripts in the background. Perhaps verify that your script takes the appropriate amount of time if you replace your python calls with this: python -c "from time import sleep; sleep(30)" Your bash script should then take 90 seconds. — sehafoc, Dec 20 '17 at 01:25
BTW, see the warning in https://docs.python.org/2/library/subprocess.html#frequently-used-arguments about why `shell=True` is dangerous -- this applies to **all** methods of invoking a subprocess that interpose a shell between the parent and its intended child, meaning `os.system()` is subject to the same dangers (and shouldn't be used for the same reason!). — Charles Duffy, Dec 20 '17 at 01:40
...to be concrete: Consider if one of your `input`s contains `$(rm -rf ~)` -- you don't want that to be executed by a shell, but that's exactly what happens if it gets substituted in place of one of your `%s` placeholders. — Charles Duffy, Dec 20 '17 at 01:40
Python running new Python instances is almost always an antipattern in its own right. Can you `import` the modules you are running and call them in-line? — tripleee, Dec 20 '17 at 05:21
If `os.system()` doesn't properly wait then you have another layer of the same problem, and the actual issue is inside these other scripts. — tripleee, Dec 20 '17 at 05:23
Re: `shell=True` maybe see also https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess — tripleee, Dec 20 '17 at 05:24
@MichaelBates, ...effectively, though, your existing code doesn't need `wait` any more than a shell script that ran those three commands in a row with no `&`s would need to use (or gain any benefit from) `wait`. Though the shell script would be more secure, assuming it's expanding `"$input"` for the parameters and not using `eval` (which using string substitution to generate shell commands does in effect). — Charles Duffy, Dec 20 '17 at 16:01
Ok thanks, ill take onboard everyones advice here and give it a shot, from what I am understanding, I should ditch this python script and call those three individual scripts directly in the shell script, ill check for the (rm -rf) and any situations where its using 'eval'. — Michael Bates, Dec 20 '17 at 19:39

seventyseven · Answer 1 · 2017-12-20T01:33:33.220

0

Use subprocess.check_call() not os.system()

subprocess.check_call() will block your main Python script's execution until the function has returned a value.

Documentation for check_call() here

The subprocess module should always be used instead of os.system() for subprocess management and execution.

edited Dec 20 '17 at 01:33

answered Dec 20 '17 at 01:27

seventyseven

41
4

This is good advice in general, but `os.system()` blocks too. – Charles Duffy Dec 20 '17 at 01:35
...which is to say, I don't see how following this advice would solve any problem the OP's question can be reasonably interpreted as asking about. – Charles Duffy Dec 20 '17 at 02:05
You're right, I forgot ``os.system()`` was blocking. Haven't used it in years. :) Thanks – seventyseven Dec 20 '17 at 02:09

score 0 · Answer 2 · answered Jan 14 '20 at 14:01

Thank you to all that helped, here is what caused my dilemma for anyone google searching this. I determined from inserting "python -c "from time import sleep; sleep(30)"" into my code that the first two python scripts were waiting as expected, the final one was not (the timer would immediately trigger after that script ran), turns out that the third python script also called another small python script that had a "&" at the end of it that was ignoring any commands to wait on it. Simply removing this & allowed all the code to run sequentially. – Michael Bates

Proper use of os.wait()?

2 Answers2