How to capture stderr in a variable and pipe stdout through

Question

I'm trying to store the headers (from stderr) of a curl response in a variable and pipe the body (from stdout) to grep.

Here's my current attempt:

{
  HEADERS=$(curl -vs $URL 2>&1 1>&3-)
  echo "$HEADERS"
} 3>&1 | grep "regex" >> filename
echo "$HEADERS"

When I run the script with bash -x script.sh I see + HEADERS='...' with the expected output, but I can't access them with either $HEADERS nor "$HEADERS" inside and outside the inline group.

The body get's piped through as expected.

So you want the `grep` output to be on terminal or somewhere else? — anishsane, Apr 21 '16 at 09:42
Any reason not to just set header as a variable, then grep the variable instead of this complicated pipe ? — 123, Apr 21 '16 at 09:51
@anishsane the `grep` output should in this case be appended to `filename`. This part works. — André Rüdiger, Apr 21 '16 at 09:59
@123 The variable contains the headers from stderr. The `grep` processes the body from stdout. — André Rüdiger, Apr 21 '16 at 10:03
Does [Store/capture stdout and stderr in different variables](https://stackoverflow.com/questions/11027679/store-capture-stdout-and-stderr-in-different-variables-bash) actually cover what you need to know? It's one level more complex because it has both streams caught in variables, but it is otherwise covering very similar territory. — Jonathan Leffler, Apr 21 '16 at 20:45

anubhava · Answer 1 · 2016-04-21T21:02:40.877

2

You lose HEADERS variable due to use of pipes which forks and runs your piped commands in a sub shell and parent shell doesn't see variable created in child shells.

You can do this via a temporary file:

{ f=$(mktemp /tmp/curl.XXXXXX); curl -vs "$URL" 2>"$f" |
grep 'regex' >> filename; HEADERS="$(<$f)"; trap 'rm -f "$f"' EXIT;}

Now HEADERS variable will be populated in parent shell itself by reading temporary file created using mktemp.

edited Apr 21 '16 at 21:02

answered Apr 21 '16 at 20:20

anubhava

761,203
64
569
643

1

Would it be a good idea to show the `rm -f "$f"` after assigning to `HEADERS`? What about interrupts, traps and clean up of the file? – Jonathan Leffler Apr 21 '16 at 20:47
Thanks, I added `trap` to clean the temporary file. – anubhava Apr 21 '16 at 20:57
Thanks for this version, which works as well. I accepted @JonathanLeffler's solution as it avoids temporary files. – André Rüdiger Apr 22 '16 at 07:41

score 2 · Accepted Answer · edited May 23 '17 at 12:09

As anubhava correctly diagnosed, the problem is that you're setting HEADERS in a sub-process, not in the main process of your shell.

You can use Bash's process substitution to avoid the problem, without using temporary files which have to be cleaned up in the event of the shell being interrupted:

HEADERS=""
{ HEADERS=$(curl -vs "$URL" 2>&1 1>&3-); } 3> >(grep "regex" > file)
echo "$HEADERS"

The grep sub-process is hidden by the process substitution. The space between 3> and >(…) is necessary.

How to capture stderr in a variable and pipe stdout through

2 Answers2