0

I have a command which runs a python script that has some prints and also returns a result to stdout. My question is how can I run this script and pipe the result to next command, without piping the prints.

For example, if my python script is like this,

def my_python_script_logic:
  print("script started")
  print("script keeps running")
  print("script finishes soon")
  sys.stdout.write("script result")

When running this in terminal,

my_python_script_command | xargs -I {} the_next_command {}

how to avoid piping the string that got printed to the next command, and only pipe the "script result"?

p.s. Not necessary to use xargs. It's just something I tried.

p.s. One interesting found, if I run this,

my_python_script_command | xargs -I {} echo {}

the echo only display the "script result".

Thus I tried this,

my_python_script_command | xargs -I {} echo {} | xargs -I {} the_next_command {}

but still not working

Andy Wang
  • 13
  • 2
  • The default action of `xargs` is to print its arguments, so `xargs echo` is pretty redundant. – tripleee Jan 25 '21 at 07:37
  • Related: [The difference between `sys.stdout.write` and `print`?](https://stackoverflow.com/questions/3263672/the-difference-between-sys-stdout-write-and-print) – kvantour Jan 25 '21 at 07:43

2 Answers2

1

This is why the operating system offers a separate output channel for diagnostic messages, called standard error.

From Python 3 code, you can use print with file=sys.stderr, though a better solution is probably to learn to use the logging module for diagnostics.

import logging


logger = logging.getLogger(__name__)

def my_python_script_logic:
  logger.debug("script started")
  logger.info("script keeps running")
  logger.info("script finishes soon")
  logger.warning("something happened")
  if something != other:
     logger.error("horrible things prevented us from delivering results")
     raise HorribleException(f"{something} != {other}")
  sys.stdout.write("script result")

One of the benefits that this brings is that you can tweak down the verbosity to discard debug and info messages by setting the logging level. Another is that logging does lazy evaluation of arguments, so you can do things like

    logging.warning("this should have happened: %s", expensive_operation())

and only have expensive_operation() be evaluated if the warning is actually printed.

tripleee
  • 175,061
  • 34
  • 275
  • 318
  • Thank you @tripleee for answering, appreciate it! But my current codebase has quite a lot print() statements, which makes it hard to update all of them to logger.xxx. Is there other way to make the "script result" return to be in a separate output channel, so that I can store the result in a variable like my_variable=$(my_python_script_command)? – Andy Wang Jan 25 '21 at 18:14
  • No, not really. Or well, you could print the actual result to `sys.stderr` but that would decidedly be very confusing. How would the calling code know which `print` statements are the result? – tripleee Jan 25 '21 at 18:48
  • Lexically replacing the diagnostic `print` statements might seem like a lot of work, but even if there are hundreds of them, separating diagnostic messages to use a separate file handle (and a different function to output them) just makes sense. My prediction is that you'll find this to have been time well spent, and probably less of it than you imagined at first. – tripleee Jan 25 '21 at 18:52
  • Hmm I see. But I found `my_python_script_command | xargs -I {} echo {}` is actually able to get that "script result". Do you know why is that, and do you think this is something we can utilize, even in a hacky way? – Andy Wang Jan 25 '21 at 22:21
0

@tripleee's answer is totally correct. Basically the codebase I'm working on is misusing print() a lot, which caused this problem. And the ideal solution is to update them to use logger.debug/info/warning.

But I still cannot do that since there are too many print() to handle. I ended up doing some hack to catch the final stdout. The solution is to preface the final stdout with a unique filterable string, e.g. "unique_prefix: ". And then in terminal just receive all stdout and filter by regular expression.

For example,

my_python_script_command | perl -ne 'if(/unique_prefix: (.*)/){print($1);}' | xargs -I {} the_next_command {}

Andy Wang
  • 13
  • 2