0

I want to run a sub process with popen that can send messages to both stdout and stderr, but that sub process will continue along it's merry way despite writing to stderr.

I want to stream stdout and stderr together so I get the output in the exact order it occurred (or was flushed I guess technically). Then, I want to log that full result set. BUT I also want to independently know if stderr is empty. If it is not, I'm going to throw an exception.

It is clear to me how I can get them separately, or merged together, but how can I perhaps do both?

BuvinJ
  • 10,221
  • 5
  • 83
  • 96
  • How do you do to merge them together? Do you mean on screen or in a variable? – JohanL Feb 05 '18 at 09:09
  • In a variable. I would also be fine with a flat file if that made it any easier. – BuvinJ Feb 05 '18 at 13:46
  • I've started looking at using some ideas from this post: https://stackoverflow.com/questions/16396873/popen-mixed-data-stream – BuvinJ Feb 05 '18 at 13:48
  • The trouble I see with those approaches is that they aren't designed for truly "fast" streams, and there doesn't seem to be a precise control for the order of events. The process I'm polling can work lightning fast and dump messages on both streams. Using arbitrary sleeps, and readline() from sdtout and stderr could give me a good approximation of the message order but it would not be perfect, the way a redirection of both streams to the same file descriptor would be. (assuming a flush occurs at the time of each write, which I *think* it does...) – BuvinJ Feb 05 '18 at 13:54
  • I would be fine with using an alternative to popen, or some combination of shell command redirection streams. Combining such messages in order is a common and typical task. I feel like there ought to be a clean and easy way to just detect if anything passed through through the stderr stream or not... – BuvinJ Feb 05 '18 at 13:59
  • Btw, one can discard the threading in those SO posts for my needs. I'm looking for a synchronous/blocking solution, not a real time ui stream. The value of those examples is just in how one can poll both streams independently in pseudo real-time and thus sort of determine the order of events. – BuvinJ Feb 05 '18 at 14:03
  • Well, there is no real order between stderr and stdout, so they will always come somewhat arbitrarely. stdout is buffered, while stderr is not (and thus not flushed in the same sense). Therefore it is hard to define a "correct" order between the two. What is generating the two streams? What are you trying to achieve? – JohanL Feb 05 '18 at 15:19
  • I'm launching the MySQL terminal client and executing `source` commands to run scripts. You can't execute `source` via MySQLdb. That does not stop on a error, it keeps right on going. Successful responses go to stdout, errors go to stderr. When reviewing the results of the script, you need to be able to see the exact order these messages came back. I have no desire to try and parse the sql scripts. I want them to be open ended without imparting rules like "use the ; command delimiter and never change your delimiter, and never use the char other than for the delimiter,etc.... – BuvinJ Feb 05 '18 at 15:28
  • In theory, I could search combined results for something like the string "error", but that would be seriously flawed. The fact that stderr was written to is far more precise for error detection. – BuvinJ Feb 05 '18 at 15:30
  • Does this mean that for an operation you will get either a stderr or a stdout and not both? And in between you have some time to ffigure out which, before the next command? Or how do you ensure the correct order between the two (partially) independent streams? – JohanL Feb 05 '18 at 17:33
  • Yes. When you run a sql script through MySQL, it writes to one stream or the other. Stdout for success indications, or query results, and stderr for errors. It outputs something to one or the other for every statement, and you can execute an infinite number of them. By comparing the results to what you ran you can figure out how the script worked. It is absolutely critical you have these in order, so you determine the one-to-one result, and how your procedural script progressed. – BuvinJ Feb 05 '18 at 17:46
  • You can't control the time between operations. It happens as fast as possible (commonly with milliseconds between them). Further, this is to allow other developers to run open ended scripts, not things that I control or want to be trying to programmatically manipulate. In sql scripts you can freely change how your subsequent statements will be parsed even. It's very common to change statement delimiters. It would be a horrible mess to try and parse or inject into the source scripts, or to parse the combined results. – BuvinJ Feb 05 '18 at 17:49
  • I need "open ended input" with "open ended output". I just want to detect if any data streamed into stderr. If it is does, after logging the combined results, I'll throw an exception to prevent subsequent scripts from running. Then, a developer can break up a sequence as they wish by submitting multiple scripts. There is no option in MySQL to stop on error. That feature was requested 10 years ago, at least, and they never provided it. This will approximate that. – BuvinJ Feb 05 '18 at 17:55
  • This is probably more difficult than it's worth, but maybe I could direct the data into a filter of some kind that could detect the source? I think I could open a named pipe and send it through that to be processed... This just seems like a lot of work and code for something that ought to be simple. – BuvinJ Feb 05 '18 at 18:11
  • @JohanL in case you're interested, I posted a conceptual answer – BuvinJ Feb 08 '18 at 01:07

1 Answers1

0

First off I solved my actual use case in large part by digging further into the process producing the streams. I was using the MySQL client, and found the right combination of command line switches would solve the real goals. I'll leave that out since it's entirely specific to that situation.

Before I came upon that, however, I did find a very useful direction to explore - the use of the program "Tee" and / or a Python source equivalent. I did not try to implement it, but is seems like a method one could use to simultaneously collect the output from both stdout and stderr, while sending just stdout to another file descriptor. If the two outputs are not the same, then that would tell you if stderr was utilized.

Check out :

https://www.computerhope.com/unix/utee.htm

And :

Python subprocess get children's output to file and terminal?

BuvinJ
  • 10,221
  • 5
  • 83
  • 96
  • I thought about `tee` while thinking of your problem. It, however, will lead to the same issues as all other soluitons. It may change the order of your streams. The stdou and stderr are independent streams. Redirecting them to the same stream does not give a fixed ordering, but depends on the current state of the OS and its other operations. Adding `tee`to the mix, may change this order and is therefore not guaranteed to give the same result. However, if you have intrinsic knowledge of the orders of `stdout` and `stderr` this can be incorporated into a working solution with or without `tee`. – JohanL Feb 08 '18 at 08:09
  • The makes perfect sense. Thanks @JohanL. At least in my use case, I found a switch for the MySQL Client, i.e. `--unbuffered` which forces an stdout/stderr flush upon each write to them, thereby addressing this exact issue (presumably for this exact reason). Of course, that wouldn't help for some other process which one had no such control... – BuvinJ Feb 08 '18 at 14:24