GNU parallel sorted stdout and stderr

Question

I've been using GNU parallel and I want to keep output order (--kepp-order), grouped by jobs (--grouped) but also with sorted stdout and stderr. Right now, the grouped options first print stdout and only after does it print stderr.

As an example, any way that these two commands give the same output?

seq 4 | parallel -j0 'sleep {}; echo -n start{}>&2; sleep {}; echo {}end'

seq 4 | parallel -j0 'sleep {}; echo -n start{}   ; sleep {}; echo {}end'

thanks,

score 2 · Answer 1 · answered Jan 07 '19 at 05:36

As per the comment to the other answer, to keep the output ordered, simply have parallel's bash invocation redirect stderr to stdout:

parallel myfunc '2>&1'

E.g.,

parallel -j8 eval \{1} -w1 \{2} '2>&1' ::: "traceroute -a -f9" traceroute6 ::: ordns.he.net one.one.one.one google-public-dns-a.google.com

score 1 · Accepted Answer · answered Oct 12 '15 at 17:44

1

You cannot do that if you still want stderr and stdout to be separated.

The reason for this is that stderr and stdout are buffered to 2 different files using buffered output.

But maybe you can explain a bit more on what you need this for. In that case there might be a solution.

answered Oct 12 '15 at 17:44

Ole Tange

31,768
5
86
104

I'm trying to parallelize several bash commands, so I placed them all inside a bash function and call that function from parallel and the output stored in a log file. So far so good, but some commands print to stdout while others to stderr! Ideally, I'd like the messages to be displayed in order so that I can go through the log file and see if anything went wrong and where... – FGV Oct 13 '15 at 12:50
2

So how about taking the commands that log to stdout and redirecting those to stderr, so stderr contains all logging information? Or simply send everything to stdout: parallel myfunc '2>&1' ::: args – Ole Tange Oct 13 '15 at 16:06

methuselah-0 · Answer 3 · 2019-06-22T17:05:17.057

Assuming that you don't have to use gnu parallel, and the main requirements are parallel execution with maintained ordered output of both stderr and stdout; we can create a solution that allows for the following example usage(plus providing return code), where you will have the results of the executions in a list, where each list element is in return a list of 3 strings: indexed as 0=stdout, 1=stderr and 2=return code.

source mapfork.sh
ArgsMap=("-Pn" "-p" "{}" "{}")
Args=("80" "google.com" "25" "tutanota.com" "80" "apa bepa")
declare -a Results=$(mapfork nmap "(${ArgsMap[*]@Q})" "(${Args[*]@Q})")

So, in order to print for example the stderr results, of the third destination ("apa bepa"), you can do:

declare -a res3="${Results[2]}"
declare -p res3
# declare -a res3=([0]=$'Starting Nmap 7.70 ( https://nmap.org ) at 2019-06-21 18:55 CEST\nNmap done: 0 IP addresses (0 hosts up) scanned in 0.09 seconds' [1]=$'Failed to resolve "apa bepa".\nWARNING: No targets were specified, so 0 hosts scanned.' [2]="0")
printf '%b\n' "${res3[1]}"

mapfork.sh is shown below. It is a bit complicated but it's parts have been explained in other answers so I won't provide the details here as well:

Capture both stdout and stderr in Bash [duplicate]

How can I make an array of lists (or similar) in bash?

#!/bin/bash
# reference: https://stackoverflow.com/questions/13806626/capture-both-stdout-and-stderr-in-bash
nullWrap(){
    local -i i; i="$1"
    local myCommand="$2"
    local -a myCommandArgs="$3"
    local myfifo="$4"
    local stderr
    local stdout
    local stdret
    . <(\
    { stderr=$({ stdout=$(eval "$myCommand ${myCommandArgs[*]@Q}"); stdret=$?; } 2>&1 ;\
               declare -p stdout >&2 ;\
           declare -p stdret >&2) ;\
      declare -p stderr;\
    } 2>&1)
    local -a Arr=("$stdout" "$stderr" "$stdret")
    printf "${i}:%s\u0000" "(${Arr[*]@Q})" > "$myfifo"
}
mapfork(){
    local command
    command="$1"
    local -a CommandArgs="$2"
    local -a Args="$3"
    local -a PipedArr
    local -i i
    local myfifo=$(mktemp /tmp/temp.XXXXXXXX)
    rm "$myfifo"
    mkfifo "$myfifo"

    local -a placeHolders=()
    for ((i=0;i<${#CommandArgs[@]};i++)); do
    [[ "${CommandArgs[$i]}" =~ ^\{\}$ ]] && placeHolders+=("$i") ;done

    for ((i=0;i<${#Args[@]};i+=0)); do
    # if we have placeholders in CommandArgs we need to take args
    # from Args to replace.
    if [[ ${#placeHolders[@]} -gt 0 ]]; then
        for ii in "${placeHolders[@]}"; do
        CommandArgs["$ii"]="${Args[$i]}"
        i+=1; done; fi
    nullWrap "$i" "$command" "(${CommandArgs[*]@Q})" "$myfifo" &
    done
    for ((i=0;i<${#Args[@]};i+=$(("${#placeHolders[@]}")))) ; do
    local res
    res=$(read -d $'\u0000' -r temp <"$myfifo" && printf '%b' "$temp")
    local -i resI
    resI="${res%%:*}"
    PipedArr[$resI]="${res#*:}"
    done
    # reference: https://stackoverflow.com/questions/41966140/how-can-i-make-an-array-of-lists-or-similar-in-bash
    printf '%s' "(${PipedArr[*]@Q})"
}

GNU parallel sorted stdout and stderr

3 Answers3