11

I'm telling jupyter to execute a python script:

!python build_database.py

When executed from the terminal, the python script prints the progress during execution. However, in the jupyter notebook, I get all output printed as a list of strings, AFTER the execution. Is there a way to see the output live?

Benni
  • 795
  • 2
  • 7
  • 20
  • 1
    More full featured support in Jupyter is to use `%run` and not `!python`. In this example, that would be`%run build_database.py`. It often gives a different, better experience depending on your system. Plus, it has more options, see [here](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-run). – Wayne Jan 28 '22 at 04:37
  • 1
    There's some settings about buffering that can be adjusted. There's a related discussion on the Jupyter Discourse Community Forum [here](https://discourse.jupyter.org/t/how-to-show-the-result-of-shell-instantly/8947?u=fomightez). – Wayne Jan 28 '22 at 04:41

2 Answers2

13

It looks like it is not possible out of the box. The output handling of shell commands is buried deep in ipython internals.

One of solutions i would recommend is to create custom magic method based on code below.

Check this answer

Based on it i created a simple magic method that you can use:

from subprocess import Popen, PIPE, STDOUT

from IPython.core.magic import register_line_magic


@register_line_magic
def runrealcmd(command):
    process = Popen(command, stdout=PIPE, shell=True, stderr=STDOUT, bufsize=1, close_fds=True)
    for line in iter(process.stdout.readline, b''):
        print(line.rstrip().decode('utf-8'))
    process.stdout.close()
    process.wait()

Or Popen can be used as context manager. So the code will be a bit more readable and reliable. See docs

Popen objects are supported as context managers via the with statement: on exit, standard file descriptors are closed, and the process is waited for.

from subprocess import Popen, PIPE, STDOUT

from IPython.core.magic import register_line_magic


@register_line_magic
def runrealcmd(command):
    with Popen(
        command, stdout=PIPE, shell=True, stderr=STDOUT, bufsize=1, close_fds=True
    ) as process:
        for line in iter(process.stdout.readline, b""):
            print(line.rstrip().decode("utf-8"))

Usage:

%runrealcmd ping -c10 www.google.com

Above code probably could be written better but for your needs it should be perfectly fine.

Kamil Niski
  • 4,580
  • 1
  • 11
  • 24
  • 1
    It doesn´t change the fact for me that the result is returned after complete execution. – Benni Sep 30 '18 at 11:08
  • I will research the problem some more and get back to you. – Kamil Niski Sep 30 '18 at 11:19
  • 1
    Maybe an important detail: The script build_database.py uses print() to output the progress I want to see in realtime – Benni Sep 30 '18 at 11:21
  • I have updated my answer, take a look if that solution satisfies you – Kamil Niski Oct 02 '18 at 06:01
  • I get: `ValueError: close_fds is not supported on Windows platforms if you redirect stdin/stdout/stderr` and with `close_fds=False` it is not realtime – Benni Oct 02 '18 at 15:34
  • so you removed `close_fds` and it is not printing in realtime? – Kamil Niski Oct 02 '18 at 15:36
  • I set `close_fds=False` and it is not printing in realtime – Benni Oct 02 '18 at 18:45
  • You can try code above with -u param on python: %runrealcmd python -u build_database.py. Moreover you should put Kamil Niski's runrealcmd code to .ipython\profile_default\startup\runcmd.py so it automatically load custom magic – o0omycomputero0o May 15 '20 at 17:17
  • I've got similar issue which alas the above approach does not resolve. In my case wanting progress reporting from the mpi example on https://emcee.readthedocs.io/en/stable/tutorials/parallel/ . The code runs alright but I don't see any output until the subprocess is complete. – Sideshow Bob Jul 15 '21 at 15:10
  • Thanks @KamilNiski for the post! This fixed a very bizarre issue with the `matplotlib_iterm2` package inside `jupyter console`: https://github.com/oselivanov/matplotlib_iterm2/pull/4 – Luke Davis Mar 03 '22 at 02:01
  • 1
    @LukeDavis great to hear that! You should also be able to use `Popen` as context manager. I've added example in my original answer. – Kamil Niski Mar 03 '22 at 10:35
3

I refer to this answer: How to show real time process command line in Jupyter Notebook?

Just execute your python script as:

%run build_database.py
Veltriuk
  • 31
  • 2