11

I am running some scripts with commands having cat pipelined with grep like:

cat file.txt | grep "pattern"

Most of the times there are no problems. But sometimes I get below error:

cat: write error: Broken pipe

So how do I find out when the command is causing this problem and why?

Vishal-L
  • 1,307
  • 1
  • 9
  • 16
  • What it the content of `file.txt`, is it a growing file? – Inian Apr 05 '18 at 05:55
  • @Inian, No it's a read-only file and is never changed any time. – Vishal-L Apr 05 '18 at 05:58
  • 5
    The reason is because the pipe is closed by `grep` when it still has some data to be read from `cat`. The signal `SIGPIPE` is caught by `cat` and it exits – Inian Apr 05 '18 at 06:05
  • 1
    @Inian, Yeah I know that can be a possible cause, but how can pipe be closed by grep before cat completes and why is this happening rarely? – Vishal-L Apr 05 '18 at 06:09
  • 7
    Not sure which version of `grep` and `bash` shell you are running. It is not easily reproducible. For possible fixes I would suggest removing `cat` usage and do just `grep "pattern" file` to avoid the pipeline altogether – Inian Apr 05 '18 at 06:11
  • If you were running `grep -n 3`, that would cause it (when there are less than 3 matches, and the last is not near the end of the file). – Charles Duffy Sep 09 '19 at 13:03
  • However, **you shouldn't ever run `cat file.txt | grep` in the first place**. It's extremely inefficient compared to `grep -e "pattern" file.txt`, or `grep -e "pattern" – Charles Duffy Sep 09 '19 at 13:03

2 Answers2

16

The reason is because the pipe is closed by grep when it still has some data to be read from cat. The signal SIGPIPE is caught by cat and it exits.

What usually happens in a pipeline is the shell runs cat in one process and grep in another. The stdout of cat is connected to the write-end of the pipe and stdin of grep to the read end. What happened was grep hit a pattern search that did not exist and exited immediately causing the read end of the pipe to be closed, which cat does not like since it has some more data to be write out to the pipe. Since the write actions happens to an other which has been closed other end, SIGPIPE is caught by the cat on which it immediately exits.

For such a trivial case, you could remove the pipeline usage altogether and run it as grep "pattern" file.txt when the file's contents are made available over the stdin of grep on which it could read from.

Inian
  • 80,270
  • 14
  • 142
  • 161
  • 1
    This depends on the version right? I've created a file with 31948800 lines of "y" and one last line with "x". When executing, `cat file | grep x`, grep waits until cat finishes. Same happens with `(echo 4; sleep 5; echo 5) | grep 5` – jinawee Aug 26 '19 at 09:25
  • 5
    "hit a pattern search that did not exist"? How does it know that without reading all the way to the end? (I need to assume that the OP's *real* use case is using `grep -q`, `grep -n`, or another argument they elided from the question). – Charles Duffy Sep 09 '19 at 13:05
-1

You can use only grep without pipe like this :

grep "pattern" file.txt

I think it's better to resolve this problem

ramzieus
  • 138
  • 11