3

I have a python script that takes several hours to complete. I usually run it as:

python myscript.py arg1 arg2 ... argN 2> log.err > log.out &

I'd like to have a shell script that runs several time the same python script (with different parameters). This script has to start the second python script ONLY when the first one has terminated and so on. I think I should use PID of the processes and iteratively check using sleep within some while loops. Any suggestions? The scripts are executed in a remote machine so I have to put them in background in order the guarantee they keep executing even when I log out.

NCL
  • 355
  • 2
  • 4
  • 12

4 Answers4

5

Instead of backgrounding each invocation of python, use a script which runs all the jobs one by one and background the script. For example:

#!/bin/sh

python myscript.py arg1 arg2 ... argN 2> log.err > log.out
python myscript.py different_args
# etc.

Make the script executable (chmod +x script.sh) and run it from the shell like ./script.sh &

Perhaps you should take a look at a tool like screen, which will keep your jobs running after you log off. If you do screen ./script.sh, then detach (ctrl-a-d) and log off, your script will continue to run. Then you can log back in later and reattach with screen -r.

Tom Fenech
  • 72,334
  • 12
  • 107
  • 141
  • The python scripts can't run at the same time. The have to be executed one by one sequentially – NCL Aug 03 '14 at 17:23
  • In my answer, that is exactly what will happen. – Tom Fenech Aug 03 '14 at 17:23
  • The problem is that I run them in a remote machine so they have to be in background to be running even when I log out – NCL Aug 03 '14 at 17:25
  • In that case, you should consider using the `screen` utility to keep an active session open when you log out, or run the entire `script.sh` in the background, as @TomFenech points out. – alexkonradi Aug 03 '14 at 17:27
  • Then just run the global script with `nohup ./script.sh &`. Nohup avoid background script to be killed at end of interactive session. – Serge Ballesta Aug 03 '14 at 17:28
  • @alexkonradi I was just in the process of suggesting screen :) – Tom Fenech Aug 03 '14 at 17:30
  • @NCL please see my update regarding the use of a tool such as `screen` to keep your job running in the background between interactive sessions. – Tom Fenech Aug 03 '14 at 17:31
1

For example, you can use cycle (make a script runner.sh with the next content:

n=0
while read -r line
do
    let n++
    echo python myscript.py $line #>log.$n.log 2>log.$n.err
done <<ARGS
a1 a2 a3 a4
b1 b2
c1 c2 c3 c4
ARGS

this will dry run your python script with args

a1 a2 a3 a4
b1 b2
c1 c2 c3 c4

sequentially, e.g. when the previous ends.

If you satisfied, remove the echo and the # from the line

    echo python myscript.py $line #>log.$n.log 2>log.$n.err

The log files will be numbered in sequence... 1..n.

Why do you want run the python script in the background? You always can send to background the above script and the whole sequence will be done in the bg.

If you need run it after the logout, use the above runner.sh as

chmod 755 runner.sh
nohup runner.sh
logout

or much better use the screen utility, and you can logout and login from another place and re-attach the session and so on... read man screen.

clt60
  • 62,119
  • 17
  • 107
  • 194
0

Make a bash script?

python first.py
python second.py
locohamster
  • 117
  • 6
0

If you want one script to run directly after the other, simply drop the trailing &. The ampersand tells bash to run the process in the background, which allows you to run multiple processes in parallel (some in the background, and some in the foreground). If you want to run them sequentially, you should run them each in the foreground, like so:

python myscript.py argX1 argX2 ... argXN 2> logX.err > logX.out
python myscript.py argY1 argY2 ... argYN 2> logY.err > logY.out

This will run the script with the argx arguments, then, after that finishes, run the same script again with the argY arguments.

If you want the next execution to run only if the first run is successful, you can use the double ampersand (&&) like so (note that the trailing backslash (\) escapes the newline, so the entire thing is actually a single command):

python myscript.py argx1 argx2 ... argN 2> logx.err > logx.out && \
python myscript.py argy1 argy2 ... argN 2> logy.err > logy.out

You can chain more than two commands with double ampersands, but for longer scripts, you should consider using set -e at the beginning, which will cause bash to stop executing the script after the first error (see http://www.davidpashley.com/articles/writing-robust-shell-scripts/#id2382181)

alexkonradi
  • 750
  • 5
  • 5