Two bash jobs executed at the same time

Question

I am using Python to execute an external program by simply using Python's "os" library:

os.system("./script1") # script1 generates several log files and the most important for me is "info.log" !
os.system("./script2") # script2 uses the output from script1

The problem is that those scripts are in 50000 element "for loop" and "script1" needs at least 2 min to finish its job (it has fixed time duration) In the first 1-2 sec I can find out whether I need the output data or not by looking in to the "info.log" file. However, as "script1" is an already compiled program and I can't modify it, I have to wait until it finishes.

I was thinking about a method in Bash that allows me to run two processes at the same time: one is to start ./script1 and the other is to monitor any change in the "info.log" file...
If "info.log" has been updated or changed in size, then the second script should terminate both processes.

Something like:

os.system("./mercury6 1 > ./energy.log 2>/dev/null & sleep 2 & if [ $(stat -c %s ./info.log) -ge 1371]; then kill %1; else echo 'OK'; fi")

– which does not work...

Please, if someone knows a method, let me know!

Answer 1

import os
import time

os.system("script1")

while True: # Now this would run in 10 sec cycles and checks the output of the logs
   try:     # And you don't have the wait the whole 2 mins to script1 to be ready
       o = open("info.log", "r") # or add os.system("script2") here
       # Add here more logs
       break
   except:
       print "waiting script1 to be ready.."
       time.sleep(10) # Modify (seconds) to be sufficient to script1 to be ready
       continue

Answer 2

I would suggest using subprocess, with a combination of this answer describing how to kill a bash script along with some method of monitoring the file, something like watchdog, or a simple polling loop.

Answer 3

You can try this one. Tell me if script1 pauses on runtime so we could try to configure it further.

#!/bin/bash

[ -n "$BASH_VERSION" ] || {
    echo "You need Bash to run this script."
    exit 1
}

set +o monitor ## Disable job control.

INFO_LOG='./info.log'

if [[ ! -f $INFO_LOG ]]; then
    # We must create the file.
    : > "$INFO_LOG" || {
        echo "Unable to create $INFO_LOG."
        exit 1
    }
fi

read FIRSTTIMESTAMP < <(stat -c '%s' "$INFO_LOG") && [[ $FIRSTTIMESTAMP == +([[:digit:]]) ]] || {
    echo "Unable to get stat of $INFO_LOG."
    exit 1
}

# Run the first process with the script.

./script1 &
SCRIPT1_PID=$!
disown "$SCRIPT1_PID"

# Run the second process.

(
    function kill_processes {
        echo "Killing both processes."
        kill "$SCRIPT1_PID"
        exit 1
    }

    [[ -f $INFO_LOG ]] || {
        echo "File has been deleted."
        kill_processes
    }

    read NEWTIMESTAMP < <(stat -c '%s' "$INFO_LOG") && [[ $NEWTIMESTAMP == +([[:digit:]]) ]] || {
        echo "Unable to get new timestamp of $INFO_LOG."
        kill_processes
    }

    [[ NEWTIMESTAMP -ne FIRSTTIMESTAMP ]] || {
        echo "$INFO_LOG has changed."
        kill_processes
    }

    sleep 4s
) &

disown "$!"

You might also want to check some other similar solutions here: linux script with netcat stops working after x hours

---

(
    function kill_processes {
        echo "Killing both processes."
        kill "$mercury6_PID"
        exit 1
    }
    while IFS= read -r LINE; do
        [[ "${LINE}" == "ejected" ]] && {  ## Perhaps it should be == *ejected* ?
            echo "$INFO_LOG has changed."
            kill_processes
        }
    done < <(tail -f "$INFO_LOG")
) &

disown "$!"

Answer 4

It has been done ! not so elegant but it works! this is the code:

#!/bin/bash

[ -n "$BASH_VERSION" ] || { echo "You need Bash to run this script." exit 1 }

# set +o monitor ## Disable job control.

INFO_LOG="./info.out"

# Run the first process with the script.

./mercury6 1 > ./energy.log 2>/dev/null & mercury6_PID=$! disown "$mercury6_PID"

# Run the second process.

( sleep 5 function kill_processes { echo "Killing both processes." kill "$mercury6_PID" exit 1 } while IFS= read -r LINE; do
[[ "${LINE}" == "ejected" ]] || [[ "${LINE}" == "complete." ]] && { echo "$INFO_LOG has changed." killall tail kill_processes } done < <(tail -f "$INFO_LOG") )

#&

#disown "$!"

If you have any recommendations you are very welcome! BTW I will try to make it with os.fork() in python as well. Thanks again!

Answer 5

See here (Google is your friend). You can make the processes background tasks with the & sign, then use wait to have the parent process wait for its children.