0

I'm seeing my content truncated when using pipe in a sequence like:

jq ... |
jq ... |
tee tee_file |
jq ... |

# content gets truncated here

head ... | sed ... | awk ... | awk ... | sed

...but everything works fine if I redirect output to intermediary file, then immediately feed that file to the downstream processes:

jq ... |
jq ... |
tee some_file |
jq ... > intermediary_file

# content remains intact here

cat intermediary_file |
head ... | sed ... | awk ... | awk ... | sed

The tee_file is not used as input in the downstream part of the above piping sequence, only in some later commands.

Using --unbuffered in jq doesn't help.

What are the reasons for such difference in behaviour, and how can I fix it?

MacOS High Sierra 10.13.6 (17G5019) GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin17)

wass rubleff
  • 326
  • 1
  • 14
  • https://stackoverflow.com/q/16664257/1030675 – choroba Apr 12 '19 at 19:04
  • @choroba Could you elaborate how exactly is my problem similar to the linked question? – wass rubleff Apr 12 '19 at 19:10
  • 1
    It's caused by `head` terminating after having successfully written the expected amount of data. – choroba Apr 12 '19 at 19:20
  • Hm, in my case the content gets truncated BEFORE `head` even starts, I double-checked that. – wass rubleff Apr 12 '19 at 19:21
  • Try using `sed` or `tac` + `tail` as shown in the linked question instead of `head`. If it helps, the problem is the same. – choroba Apr 12 '19 at 19:23
  • 2
    What do you mean by "content"? What's being truncated? Are you talking about the data in `tee_file`? The output to the terminal? Data passed in the pipe? – that other guy Apr 12 '19 at 19:32
  • @thatotherguy The output of the last JQ preceding the comment gets truncated when sent as input to the HEAD immediately following the comment. – wass rubleff Apr 12 '19 at 19:40
  • 1
    without a tiny set of sample data that illustrates your problem, we are only guessing. If the `head` thing isn't your problem, its something else in your code, not in the utilities you are calling ;-) . right? Good luck! – shellter Apr 12 '19 at 19:41
  • @choroba Surprisingly, it works. Do I undestand correctly that `sed -n 'p'` is the easiest/safest way to send input to output as it is? – wass rubleff Apr 12 '19 at 19:42
  • `"The output of the last JQ preceding the comment"` Do you really have a comment embedded inline with a bunch of pipes? That won't work. Your code must be `jq ... | jq ... | tee some_file | jq ... | head ... | sed ... | awk ... | awk ... | sed` . Good luck. – shellter Apr 12 '19 at 19:43
  • 1
    @wassrubleff: `cat` is probably even simpler and faster. – choroba Apr 12 '19 at 19:47
  • @choroba That's the problem: `cat` didn't solve the problem. – wass rubleff Apr 12 '19 at 19:51
  • 2
    @wassrubleff The purpose of `head` is to truncate the stream and stop the input pipe. How would you be able to tell the difference between `jq` output being truncated *before* `head`, and `jq` output being truncated *by* `head`? If you can post a MCVE, that would make everything much easier. – that other guy Apr 12 '19 at 19:59
  • 2
    Earlier you said that "the content gets truncated BEFORE head even starts". Are you assuming that in your code, first `jq` runs, then another `jq` runs, then `tee` runs, etc? This is not true. All the elements run in parallel. If the last `sed` stops immediately, then the first `jq` will stop immediately. This is just like a water pipe, where plugging the output end will immediately stop water from entering the inlet. – that other guy Apr 12 '19 at 20:06
  • 3
    As an aside, any logical structure that involves `sed ... | awk ... | awk ... | sed` is an almost certainly offense against man and God and should be taken out and shot. – Paul Hodges Apr 12 '19 at 20:56

0 Answers0