4

I have a bash/zsh command with multiple pipes | that fails when using set -o pipefail. For simplicity assume the command is

set -o pipefail; echo "123456" | head -c2 | grep 5 | cat

How do I quickly find out which command is the first to fail and why? I know I can check the exit code, but that doesn't show which part of the pipeline failed first.

Is there something simpler than the rather verbose check of building up the pipeline one by one and checking for the first failing exit code?

Edit: I removed the contrived code example I made up as it confused people about my purpose of asking. The actual command that prompted this question was:

zstdcat metadata.tsv.zst | \
tsv-summarize -H --group-by Nextclade_pango --count | \
tsv-filter -H --ge 'count:2' | \
tsv-select -H -f1 >open_lineages.txt
Cornelius Roemer
  • 3,772
  • 1
  • 24
  • 55
  • Possible duplicate or highly related: [Pipe output and capture exit status in Bash](https://stackoverflow.com/q/1221833/7939871) – Léa Gris Feb 08 '23 at 17:59
  • @LéaGris I'd say highly related not duplicate, as most solutions don't work (they look up exact position of PIPESTATUS rather than [@]), it's only about exit status of 1 rather than all pipes. But yes this is where I found the answer so worth linking. This question is more similar (though also asking only for 1 pipe): https://unix.stackexchange.com/questions/14270/get-exit-status-of-process-thats-piped-to-another – Cornelius Roemer Feb 08 '23 at 18:09
  • Really, if you care about the failure of the parts, you probably shouldn't use a pipeline in the first place. `ls ... | grep foo` is *certainly* not a pipeline you should be writing. – chepner Feb 08 '23 at 20:13
  • Is your real question, "How do I check if there is a file whose name matches a regular expression?" – chepner Feb 08 '23 at 20:14
  • @chepner No not at all. My problem arose in the pipeline `zstdcat build/open_metadata.tsv.zst | tsv-summarize -H --group-by Nextclade_pango --count | tsv-filter -H --ge 'count:2' build/open_lineages.tsv | tsv-select -H -f1 >build/open_lineages.txt`. I will get rid of the example, it's contrived and was supposed to be a minimal reproducible example. It seems to make people go in a different direction so I'll edit it away. – Cornelius Roemer Feb 08 '23 at 20:45
  • @chepner See above for the original command. I was looking for a quick way of debugging that pipeline that was part of a snakemake workflow. – Cornelius Roemer Feb 08 '23 at 21:11

2 Answers2

5

In bash, use echo "${PIPESTATUS[@]}" right after the command to get the exit status for each component in a space separated list:

#!/bin/bash
$ set -o pipefail; echo "123456" | head -c2 | grep 5 | cat

$ echo ${PIPESTATUS[@]}
0 0 1 0

Beware zsh users, you need to use the lower case pipestatus instead:

#!/bin/zsh
$ set -o pipefail; echo "123456" | head -c2 | grep 5 | cat

$ echo $pipestatus
0 0 1 0

In fish you can also simply use echo $pipestatus for the same output.

Cornelius Roemer
  • 3,772
  • 1
  • 24
  • 55
0

${PIPESTATUS[@]} right after is the answer you were looking for. However, I want to advise on the first example. It's a good habit to anticipate error, so instead of testing after you should have check the path prior everything.

if [ -d "/nonexistent_directory" ]; then
  # here pipe shouldn't fail to grep
  # ...unless there's something wrong with "foo"
  # ...but "grep" may be a failure if the pattern isn't found
  if ! ls -1 "/nonexistent_directory" | grep 'foo' ; then
    echo "The command 'grep foo' failed."
#  else
#    echo "The pipeline succeeded."
  fi
else
  echo "The command 'ls /nonexistent_directory' failed."
fi

Whenever possible, avoid greping ls output in script, that' fragile...

gildux
  • 448
  • 6
  • 12
  • Thanks for the feedback - the code I gave was contrived and not what I was writing in production. It was just meant as a simple reproducible example of how one could check it. I've removed it and replaced with the actual code that prompted this question. Sorry for making your answer no longer be applicable. – Cornelius Roemer Feb 08 '23 at 20:56
  • 1
    No worry :) The main idea is still applicable: be sure the file `metadata.tsv.zst` exists and is the right file type before calling `zstdcat` on it. After that any faillure in the pipes will be related to the TSV. If you plan to do such commands chain many times or have to reuse the result elsewhere, maybe check the file is well formatted earlier. That was the considerations I was trying to share :) – gildux Feb 09 '23 at 01:49