How to process output line by line AND save the return status?

Question

First, hello to stackoverflow community. I have learnt a lot thank to very clear questions and professional replies. I never needed to ask question because there is always someone who has asked the same question before.

But not today. I haven't found any solution to my problem, and I beg for your help. I need to process the output of a function line by line in order to update a log online. I am working with bash.

The following block works pretty well:

convertor some parameters | while read line
do
    if [ "${line:0:14}" != "[informations]" ]
    then
        update_online_log "${line}"
    fi
done

But convertor may exit with different status. And I need to know what was the exit status. The code below doesn't work as it gives me the exit status of the last executed command (update_online_log).

convertor some parameters | while read line
do
    if [ "${line:0:14}" != "[informations]" ]
    then
        update_online_log "${line}"
    fi
done
exit_status=$?

The code below should work (I haven't tried it yet):

convertor some parameters > out.txt
exit_status=$?
while read line
do
    if [ "${line:0:14}" != "[informations]" ]
    then
        update_online_log "${line}"
    fi
done < out.txt
rm out.txt

But if I use this, the online log will be updated at the end of the conversion. Conversion may be a very long process, and I want to keep users updated while the conversion is in progress.

Thank you in advance for your help.

Wow, that was fast! Thank you very much it is exactly what I needed! — Slagt, Jan 08 '14 at 16:58

score 1 · Accepted Answer · answered Jan 08 '14 at 17:12

The PIPESTATUS array may be helpful for you: it saves the exit statuses of each component of the previous pipeline:

$ (echo a; exit 42) | (cat; echo b; exit 21) | (cat; echo c; exit 3) | { cat; echo hello; }
a
b
c
hello
$ echo "${PIPESTATUS[*]}"
42 21 3 0

That array is pretty fragile, so if you want to do stuff with it, immediately save it to another array:

$ (echo a; exit 42) | ... as above
$ ps=( "${PIPESTATUS[@]}" )
$ for i in "${!ps[@]}"; do echo "$i  ${ps[$i]}"; done
0  42
1  21
2  3
3  0

score 0 · Answer 2 · answered Jan 08 '14 at 16:56

Add set -o pipefail at the top of the script. From the documentation:

the return value of a pipeline is the status of the last command to exit with a non-zero status, or zero if no command exited with a non-zero status

That means it's zero if all commands succeed and non-zero if any command fails.

Just check the exit status after the done, that should work. Test it with

convertor some parameters | false | while read line
...

No line should be processed and the exit code should be 1.

score 0 · Answer 3 · answered Jan 08 '14 at 17:49

As a POSIX-compliant solution, and to show that your approach was close to working, you can use a named pipe instead of a regular file.

mkfifo out.txt
while read line; do
    if [[ $line != \[informations\]* ]]; then
        update_online_log "$line"
    fi
done < out.txt &

convertor some parameters > out.txt

rm out.txt

This code creates the named pipe, then runs the loop that consumes it in the background. It will block waiting for data from converter. Once converter exits (and you can get its exit code at that time), out.txt will be closed, the while loop process will exit, and you can remove the named pipe.

How to process output line by line AND save the return status?

3 Answers3