2

I'm using tqdm in Python to display console progress bars.

I have a function from another library that occasionally writes to both stdout and stderr inside the tqdm loop. I cannot hack the source code of that function.

While this doc shows how to redirect sys.stdout, it doesn't easily generalize to stderr because I can pass only one of stdout or stderr to the file= parameter in tqdm's __init__. Please refer to this question and its accepted answer for the minimal code that illustrates the problem.

How do I redirect both stdout and stderr together?

Ainz Titor
  • 1,497
  • 1
  • 15
  • 22

1 Answers1

5

Python provides some helpers for you in the standard libraries, look in contextlib:

>>> import io, sys
>>> from contextlib import redirect_stdout, redirect_stderr
>>> from tqdm import tqdm
>>> def foo():
...     print('spam to stdout')
...     print('spam to stderr', file=sys.stderr)
...     
>>> out = io.StringIO()
>>> err = io.StringIO()
>>> with redirect_stdout(out), redirect_stderr(err):
...     for x in tqdm(range(3), file=sys.__stdout__):
...         print(x)  # more spam
...         foo()
...         
100%|██████████| 3/3 [00:00<00:00, 29330.80it/s]
>>> out.getvalue()
'0\nspam to stdout\n1\nspam to stdout\n2\nspam to stdout\n'
>>> err.getvalue()
'spam to stderr\nspam to stderr\nspam to stderr\n'
wim
  • 338,267
  • 99
  • 616
  • 750
  • Thanks for your answer, but it doesn't work this easily at all. If you look at the example links I posted, you'll see that tqdm interacts with sys.stdout/stderr in a very sophisticated way. – Ainz Titor Aug 16 '17 at 21:19
  • In that case, could you please produce an [MCVE](https://stackoverflow.com/help/mcve). – wim Aug 16 '17 at 21:20
  • The MCVE is in [this question](https://stackoverflow.com/questions/36986929/redirect-print-command-in-python-script-through-tqdm-write) I linked. It also contains a solution on how to redirect stdout, which works well in my experiment. All I want is to redirect both out and err with that code. – Ainz Titor Aug 16 '17 at 21:24
  • The answer you linked just looks like a bad way of doing what I've shown above, except for stdout only. The contextlib redirects works for me, capturing everything from `tqdm` correctly. If it doesn't work for you, please describe how (i.e., you still need an MCVE). – wim Aug 16 '17 at 21:34
  • No, as I said, it's not a simple redirection. You can't just save stdout/stderr at the beginning and restore them at the end. Tqdm is doing a lot under the hood. The answer I linked is also in the [official tqdm doc](https://github.com/tqdm/tqdm#redirecting-writing). Your solution captures everything in strings, which is not what a progress bar is meant to be. – Ainz Titor Aug 16 '17 at 23:13
  • The progress bar needs to update itself without being broken while the other function is printing. It's hard to describe without a video, if you know what I mean. Writing everything to string defeats the whole purpose of console progress bar. – Ainz Titor Aug 16 '17 at 23:16
  • So for the progress bar output *not* to be captured, but the output from the underlying worker function(s) (on both stdout and stderr) to be captured. Is that what you're looking for? It's not clear from reading your question whether the tqdm instance is under your control or not. – wim Aug 16 '17 at 23:30