Howto do floating point compasiosn in an if-statement within a GNU parallel block?

Question

I want to run a batch process in parallel. For this I pipe a list to parallel. When I've an if-statement, that compares two floating point numbers (taken form here), the code doesn't run anymore. How can this be solved.

LIMIT=25

ps | parallel -j2 '   
    echo "Do stuff for {} to determine NUM"    
    NUM=33.3333 # set to demonstrate

    if (( $(echo "$NUM > $LIMIT" | bc -l) )); then
        echo "react..."
    fi

    echo "Do stuff..."
    '

Prints:

Do stuff for \ \ PID\ TTY\ \ \ \ \ \ \ \ \ \ TIME\ CMD to determine NUM
Do stuff...
(standard_in) 2: syntax error
#... snipp

Add a shebang and then paste your script there: http://www.shellcheck.net/ — Cyrus, Jun 02 '18 at 16:55
Depending on what you are doing and the overhead of running external programs like `bc`, it may be faster to just do everything serially in a language with proper floating-point support. — chepner, Jun 02 '18 at 18:54
In my real code, I call an external (serial) program that runs for 2-3 minutes (instead of `echo "react..."`). Is there no way to get this done? Do I need to put the parallel stuff in an extra shell script? — dani, Jun 02 '18 at 20:59

score 3 · Answer 1 · edited Jun 03 '18 at 01:28

LIMIT is not set inside parallel shell. Running echo "$NUM > $LIMIT" | bc -l exapands to echo "123 > " | bc -l which results in syntax error reported by bc. You need to export/pass/put it's value to the shell run from inside parallel. Try this:

LIMIT=25
ps | parallel -j2 '   
    LIMIT="'"$LIMIT"'"
    echo "Do stuff for {} to determine NUM"    
    NUM=33.3333 # set to demonstrate

    if (( $(echo "$NUM > $LIMIT" | bc -l) )); then
        echo "react..."
    fi

    echo "Do stuff..."
    '

Or better use env_parallel, designed for such problems.

Side note: GNU parallel was designed for executing jobs in parallel using one or more computers. For scripts running on one computer it is better to stick with the xargs command, which is more commonly available (so you don't need to install some package each time you move your script to another machine).

Re: side note. Have a look at `--embed` (from 20180322). – Ole Tange Jun 03 '18 at 07:29 — Ole Tange, Jun 03 '18 at 07:29

score 1 · Accepted Answer · answered Jun 03 '18 at 07:38

While GNU Parallel is designed to deal correctly with commands spanning multiple lines, I personally find that hard to read. I prefer using a function:

doit() {
  arg="$1"
  echo "Do stuff for $a to determine NUM"    
  NUM=33.3333 # set to demonstrate

  if (( $(echo "$NUM > $LIMIT" | bc -l) )); then
      echo "react..."
  fi

  echo "Do stuff..."
}
export -f doit

LIMIT=25
export LIMIT

ps | parallel -j2 doit

Instead of the exports you can use env_parallel:

ps | env_parallel -j2 doit

If your environment is too big, use env_parallel --session before starting:

#!/bin/bash

env_parallel --session

# Define functions and variables _after_ running --session
doit() {
  [...]
}
LIMIT=25

ps | env_parallel -j2 doit

Howto do floating point compasiosn in an if-statement within a GNU parallel block?

2 Answers2