parralelizing a part of a script shell

Question

#!/bin/bash
for tracelength in 10 20 50 100 ; do
    step=0.1
    short=0
    long=1
    for firstloop in {1..10}; do
        ratio=0
        for secondloop in {1..10} ; do
            for repeat in {1..20} ; do
               echo $tracelength $short $long $ratio >results.csv
               python3 main.py "$tracelength" "$short" "$long" "$ratio" > file.smt2                     
               /usr/bin/time /Users/Desktop/optimathsat-1.5.1-macos-64-bit/bin/optimathsat < file.smt2 > results.csv
            done
            ratio=$(echo "scale=10; $ratio + $step" | bc)
        done
        short=$(echo "scale=10; $short + $step" | bc)
        long=$(echo "scale=10; $long - $step" | bc)
    done
done

I want to parallelize the inside loop (repeat).

I have installed GNU parallel and I know some of the basics but because the loop has more than one command I have no idea how I can parallelize them.

I transfered the content of the loop to another script which I guess is the way to go but my 3 commands need to take the variables (tracelength, ratio, short, long) and run according to them. any idea how to pass the parameters from a script to a subscript. or do you maybe have a better idea of parallelization?

I am editing the question because I used the answer below but the now my execution time is always 0.00 regadless of how big file.smt2 is.

this is the new version of code:

 #!/bin/bash
    doone() {
        tracelength="$1"
        short="$2"
        long="$3"
        ratio="$4"
        #echo "$tracelength $short $long $ratio" >> results.csv
        python3 main.py "$tracelength" "$short" "$long" "$ratio" >> file.smt2
        gtime -f "%U" /Users/Desktop/optimathsat-1.5.1-macos-64-bit/bin/optimathsat < file.smt2
    }
    export -f doone
    step=0.2
    parallel doone \
             ::: 200 300 \
             :::: <(seq 0 $step 0.5) \
             ::::+ <(seq 1 -$step 0.5) \
             :::: <(seq 0 $step 0.5) \
             ::: {1..5} &> results.csv

Execute this script parallel mean you going to spawn 8000 process in sort amount of time. That might not will be that fast you like it to be. You can put the repeat loop content into a function and execute that in the background inside the for loop. Also your results.csv and file.smt2 might be messed up because processes will write it as they go, like related records might not end up next to each other. — lw0v0wl, Apr 02 '19 at 01:04
@Edvin ah, I am executing this on a server with 200cores and I want to take advantage of that, in this case, results.csv can be out of order because at the end my goal is calculating the average of all results. But file.smt2 definitely should be in order, how do I get that? — basel117, Apr 02 '19 at 07:26
Your problem could be that you use the file `file.smt2` for all jobs running in parallel. It will be overwritten by each job. In the answer this is avoided by using a pipe. — Ole Tange, Apr 04 '19 at 15:48

score 1 · Answer 1 · answered Apr 02 '19 at 10:13

In your original code you overwrite results.csv again and again. I assume that is a mistake and that you instead want it combined into a big csvfile:

doone() {
    tracelength="$1"
    short="$2"
    long="$3"
    ratio="$4"
    echo "$tracelength $short $long $ratio"
    python3 main.py "$tracelength" "$short" "$long" "$ratio" |
        /usr/bin/time /Users/Desktop/optimathsat-1.5.1-macos-64-bit/bin/optimathsat
}
export -f doone
step=0.1
parallel doone \
         ::: 10 20 50 100 \
         :::: <(seq 0 $step 0.9) \
         ::::+ <(seq 1 -$step 0.1) \
         :::: <(seq 0 $step 0.9) \
         ::: {1..20} > results.csv

If you want a csvfile per run:

parallel --results outputdir/ doone \
         ::: 10 20 50 100 \
         :::: <(seq 0 $step 0.9) \
         ::::+ <(seq 1 -$step 0.1) \
         :::: <(seq 0 $step 0.9) \
         ::: {1..20}

If you want a csv file containing the arguments and run time use:

parallel --results output.csv doone \
         ::: 10 20 50 100 \
         :::: <(seq 0 $step 0.9) \
         ::::+ <(seq 1 -$step 0.1) \
         :::: <(seq 0 $step 0.9) \
         ::: {1..20}

I have used this solution and it works in a sens that I don't get the warnings anymore. but the execution time in my **results.csv** file is always 0.00 regardless of how big is **file.smt2** — basel117, Apr 04 '19 at 13:06

parralelizing a part of a script shell

1 Answers1