Write parts of stdout to more than one file at a time in Bash

Question

I've got a long running process that I'd like to capture parts of its output to various files as it happens.

I thought I could write something to:

output to stdout as it happens
capture all output in file /tmp/
write all lines that match 3 to file /tmp/2

$ output_data_constantly | tee /tmp/1 | grep 3 > /tmp/2

but it doesn't seem to work - nothing is output to stdout

Any suggestions?

output_data_constantly:

#!/usr/bin/env bash
for i in {1..10000}; do
  echo $i ;
  sleep 0.25;
done

Note that I'm on a mac, and my Bash version is:

$ bash --version                                                                                                                                                                      11:53:30
GNU bash, version 4.4.19(1)-release (x86_64-apple-darwin17.3.0)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Because you are redirecting grep result to `/tmp/2` – SMA Oct 08 '19 at 14:08 — SMA, Oct 08 '19 at 14:08

oguz ismail · Accepted Answer · 2019-10-08T16:14:27.380

1

grep consumes tee's output, so nothing is left to the tty. That can be amended using a process substitution, but still nothing will be written to /tmp/2 until output_data_constantly stops. You need to run grep in line buffered mode, and for that you need GNU grep, e.g:

output_data_constantly | tee /tmp/1 >(grep --line-buffered 3 >/tmp/2)

edited Oct 08 '19 at 16:14

answered Oct 08 '19 at 14:09

oguz ismail

1
16
47
69

thanks for the suggestion - I tried this out, but it doesn't seem to work for me in either case. I've tried running it in `bash` and `zsh`, and in both cases, stdout is the full output, but `/tmp/2` never gets anything written to it. I am on a mac, and my bash version is `GNU bash, version 4.4.19(1)-release (x86_64-apple-darwin17.3.0) ` – Brad Parks Oct 08 '19 at 14:53
@Brad are you sure your command writes everything to stdout (not stderr), and grep matches anything? – oguz ismail Oct 08 '19 at 15:33
The way I'm trying it is to actually just run the `output_data_constantly` example, so not even my real program yet - when I do so, nothing goes to stdout using either of the above approaches, in bash or zsh. But if I just run `output_data_constantly | grep 3`, I get what you'd expect, a `3` almost immediately, then more numbers with 3's in them as time passes by – Brad Parks Oct 08 '19 at 15:41
I used `seq 100` in stead of `output_data_constantly` and it worked just right. Maybe it's a buffering issue, I don't really know – oguz ismail Oct 08 '19 at 15:49
1

ahh... maybe it is, as it works if I try it that way! – Brad Parks Oct 08 '19 at 15:57
1

you're right - it works if I add that to the grep... thanks! – Brad Parks Oct 08 '19 at 19:10

keithpjolley · Answer 2 · 2019-10-08T16:10:19.820

1

$ output_data_constantly | tee /tmp/1 | grep 3 | tee /tmp/2

You didn't give the output a chance to make it to the screen. Put in another tee to send output to /tmp/2 and stdout instead of all to /tmp/2.

Also, note that you may not get output in "real-time" since it may get buffered along the way.

Example:

$ (yes | nl | head -10000) | tee /tmp/1 | grep 3 | tee /tmp/2 | wc -l
    3439
$ wc -l /tmp/1 /tmp/2
   10000 /tmp/1
    3439 /tmp/2
   13439 total

Maybe try line buffering grep if you are wanting output before the buffer is full. On my computer that's done with --line-buffered but may be different on yours.

edited Oct 08 '19 at 16:10

answered Oct 08 '19 at 14:09

keithpjolley

2,089
1
17
20

hey.... thanks for the feedback! I've tried this, but it doesnt seem to update. I've tried running the above and waiting for a minute, but nothing shows up in `/tmp/2` at all..... and no output shows up in the terminal either.... – Brad Parks Oct 08 '19 at 14:22
(editing original answer since code in comments is not a feature) – keithpjolley Oct 08 '19 at 16:06

Brad Parks · Answer 3 · 2019-10-08T19:08:31.063

Here's another way I've found to do it, which may not be perfect, but seems to work for my use case. Maybe this wouldn't work well for long running processes with lots of output?

#!/usr/bin/env bash
rm -f /tmp/2
rm -f /tmp/1

while IFS= read -r line; do 
  echo "$line" | grep 3 >> /tmp/2
  echo "$line" | tee -a /tmp/1
done < <(output_data_constantly)

Note: idea pulled from an answer to this similar question on stackoverflow

Write parts of stdout to more than one file at a time in Bash

3 Answers3