0

The input file is CASH_FORECAST_STATUS.TXT and contains the folowing

$ cat CASH_FORECAST_STATUS.TXT    
    TEST-CASH-FCST-1-|CASHFC|EFT Format via FG|382
    TEST-CASH-FCST-2-|CASHFC|EFT Format via FG|654
    TEST-CASH-FCST-3-|CASHFC|EFT Format via FG|234
    TEST-CASH-FCST-4-|CASHFC|EFT Format via FG|123
    TEST-CASH-FCST-5-|CASHFC|EFT Format via FG|789
    TEST-CASH-FCST-6-|CASHFC|EFT Format via FG|334
    TEST-CASH-FCST-7-|CASHFC|EFT Format via FG|125

I was requested to change the following script so it would be able to process 5 lines from the status file at a time, and when one job completes successfully, to add another to the list. The idea is no more than 5 jobs in Autosys can run at a time.

#!/bin/sh

MYSTATUSFILE=$1; export MYSTATUSFILE
MYWRKDIR=$AUTOUSER/output; export MYWRKDIR
MYWRKTYPE=`echo $MYSTATUSFILE | cut -d\_ -f2`; export MYWRKTYPE
MYPSMACHINE=FSM; export MYPSMACHINE;    

MYPID=$$
#echo $MYPID
echo $MYWRKTYPE
TODAYS_DATE_TIME=`date +%Y%m%d.%H%M`; export TODAYS_DATE_TIME
cp $MYWRKDIR/$MYSTATUSFILE $MYWRKDIR/$MYSTATUSFILE.$TODAYS_DATE_TIME
cp $MYWRKDIR/$MYSTATUSFILE $MYWRKDIR/$MYSTATUSFILE.TRIM

cat $MYWRKDIR/$MYSTATUSFILE | cut -d\| -f1-2 >$MYWRKDIR/$MYWRKTYPE_WORK.$MYPID
for LINE in `cat $MYWRKDIR/$MYWRKTYPE_WORK.$MYPID`; do
    PROGNAME=`echo $LINE | cut -d\| -f1`
    PROGRUNCTL=`echo $LINE | cut -d\| -f2`
    echo $PROGNAME 
    echo $PROGRUNCTL
# Turns out some jobs have no dash at the end but rest of the PROGNAMEs do. SO we adjust the name for setting the job name

    case $PROGNAME in
                CHKX00)
                  PROGNAMEEXT="$PROGNAME"-
                  PROGNAMEVAR="$PROGNAME"-GVAR
                  ;;
                CHKX0S)
                  PROGNAMEEXT="$PROGNAME"-
                  PROGNAMEVAR="$PROGNAME"-GVAR
                  ;;
                *)
                  PROGNAMEEXT="$PROGNAME"
                  PROGNAMEVAR="$PROGNAME"GVAR
                  ;;
        esac
        echo $PROGNAME $PROGNAMEEXT $PROGNAMEVAR
        sendevent -E SET_GLOBAL -G"$PROGNAMEVAR=$PROGRUNCTL"
    sleep 10
    sendevent -E FORCE_STARTJOB -J CM_CF-"$PROGNAMEEXT""$MYWRKTYPE"_"$MYPSMACHINE"_B
# Now check for success status of this job
    sleep 20 
    CHK_STATUS=`autostatus -J CM_CF-"$PROGNAMEEXT""$MYWRKTYPE"_"$MYPSMACHINE"_B`
    echo $CHK_STATUS
#Loop until CHK_STATUS is SUCCESS or FAILURE or TERMINATED

    until [ "$CHK_STATUS" = "SUCCESS" ] 
    do
            case $CHK_STATUS in
                    INACTIVE)
                        sleep 20 
                        CHK_STATUS=`autostatus -j CM_CF-"$PROGNAMEEXT""$MYWRKTYPE"_"$MYPSMACHINE"_B`
                        ;;      
                    ACTIVATED)
                        sleep 20 
                        CHK_STATUS=`autostatus -j CM_CF-"$PROGNAMEEXT""$MYWRKTYPE"_"$MYPSMACHINE"_B`
                        ;;      
                        INITIATED)
                        sleep 20 
                        CHK_STATUS=`autostatus -j CM_CF-"$PROGNAMEEXT""$MYWRKTYPE"_"$MYPSMACHINE"_B`
                        ;;
                        STARTING)
                        sleep 20 
                        CHK_STATUS=`autostatus -j CM_CF-"$PROGNAMEEXT""$MYWRKTYPE"_"$MYPSMACHINE"_B`
                        ;;
                        RUNNING)
                        sleep 20 
                        CHK_STATUS=`autostatus -j CM_CF-"$PROGNAMEEXT""$MYWRKTYPE"_"$MYPSMACHINE"_B`
                        ;;
                        FAILURE)
                        echo $PROGNAME has a FAILURE status
                        exit 1
                        ;;
                        TERMINATED)
                        echo $PROGNAME has a TERMINATED status
                        exit 2
                        ;;
                esac
        done
# Reached here because of SUCCESS or FAILURE or TERMINATED only
        case $CHK_STATUS in
            FAILURE)
                        echo $PROGNAME has a FAILURE status
                        exit 1
                        ;;
                        TERMINATED)
                        echo $PROGNAME has a TERMINATED status
                        exit 2
                        ;;
            SUCCESS)
            sed /$LINE/d $MYWRKDIR/$MYSTATUSFILE >/$MYWRKDIR/$MYSTATUSFILE.TRIM
                diff $MYWRKDIR/$MYSTATUSFILE.TRIM $MYWRKDIR/$MYSTATUSFILE
                mv $MYWRKDIR/$MYSTATUSFILE.TRIM $MYWRKDIR/$MYSTATUSFILE
                        ;;
                esac
done

This script currently will start 1 job in Autosys and wait for a success to continue to the next line from the input file, CASH_FORECAST_STATUS.TXT

Ultimately it will be starting the following jobs at the same time: CM_CF-TEST-CASH-FCST-1-FORECAST_FSM_B thru CM_CF-TEST-CASH-FCST-5-FORECAST_FSM_B. When one of them completes, it should start CM_CF-TEST-CASH-FCST-6-FORECAST_FSM_B and monitor for success.

I'm not sure if I need to create another file to pass job names to for monitoring and give the file a limit, or if I can use a counter. This request is miles over my head as far as scripting goes.

I can answer questions, but I'm having a hard time asking this well enough for people to understand.

Emile
  • 185
  • 1
  • 9
  • 3
    A search for "parallelizing a bash script while limiting the number of jobs" shows some potentially helpful articles: https://stackoverflow.com/questions/38774355/how-to-parallelize-for-loop-in-bash-limiting-number-of-processes https://stackoverflow.com/questions/38160/parallelize-bash-script-with-maximum-number-of-processes https://stackoverflow.com/questions/1537956/bash-limit-the-number-of-concurrent-jobs – that other guy Nov 03 '17 at 17:19
  • 2
    I'd refactor this code such that it only does the part inside the `for`. Then, write a wrapper script that calls `parallel` (or similar) to run the newly-refactored in parallel batches. Otherwise, see this for ideas on limiting concurrent jobs: https://stackoverflow.com/q/1537956/2908724 – bishop Nov 03 '17 at 17:28
  • 2
    [Don't Read Lines With For](http://mywiki.wooledge.org/DontReadLinesWithFor). Letting `xargs -d $'\n' -P 5` or GNU parallel consume your input for you is going to be half the job. – Charles Duffy Nov 03 '17 at 17:31

0 Answers0