0

I wrote a very convoluted, very hackjob PHP-cli script that receives and parses JSON of changeable structure, depth and content. For that reason at that time I found it easiest to do the parsing using PHP's shell_exec() and cat | jq | grep.

Sometimes, rarely, on certain input it gives me the message Error: writing output failed: Broken pipe, which is the last message I see in cli output before the script dies. However, even when it does do that, the data is still parsed out correctly, for all the little good it does me.

I isolated the problematic piece of code to:

$jq1='cat '.$randfile.' | jq \'.\' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}"';

$jq1=trim(shell_exec($jq1));

And tried to debug it by seeing what it executes. The first line is the shell_exec argument, echoed before execution, the second line is the result of shell_exec.

Command: cat 5ca15f21.json | jq '.' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}";
Result: standalone

Command: cat 5ca59379.json | jq '.' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}";
Result: season
Error: writing output failed: Broken pipe

Command: cat 5ca7d271.json | jq '.' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}";
Result: extended

Command: cat 5ca7d7a8.json | jq '.' | grep "\[" -m 1 | grep -Po --color=none "\w{3,15}";
Result: season

(I have seen the error of my lazy ways and will be rewriting that whole section, but back then I was young and inexperienced and impatient. I'd still like to understand what's going wrong where why.)

  1. What would make it do that sometimes? The input is always jq's pretty-printed JSON, of varying structure.

  2. Even if it does get the broken pipe message, the necessary value is still parsed out and stored in the variable. What causes it to die then? I would like to know for the future if there's a way to make PHP disregard the [non-fatal] error and go on executing.

  3. Why does the shell command that produces the broken pipe error message in shell_exec behave no differently when invoked manually in bash? Where is the broken pipe and what makes it so broken?

Dan
  • 79
  • 9
  • I'm pretty sure it's because `grep -m 1` only reads until it gets to the first match, but `jq '.'` has more to say, and gets upset when there's nobody listening. See ["sed: couldn't write 26 items to stdout: Broken pipe"](https://stackoverflow.com/questions/53697280/sed-couldnt-write-26-items-to-stdout-broken-pipe) for a very similar error. – Gordon Davisson May 16 '19 at 16:15

1 Answers1

0

Edit:

The cat command could be eliminated if this is a bash command to be executed. I would write it like this.

$jq1=grep -Po --color=none "\w{3,15}"|"\[" -m 1 file.txt | jq \'.\';
$jq1=trim(shell_exec($jq1));

I combined your redundant grep commands and added file to the grep command and only piped the jq command.

goose goose
  • 86
  • 3
  • 15
  • 1. I added a semicolon to properly end that line of php code, this was only added as I was prepping the post. Thank you. 2 and 3. What I essentially have is php code that produces a bash command sequence that is executed by shell_exec—so, for example, $randfile is filled in before the bash sequence is generated. The code works, and I'm not sure how to implement your suggestions, and how that helps the broken pipe problem. – Dan May 16 '19 at 05:40