Print realtime output of subprocess in jupyter notebook

Question

In my file 'wrapper.py' I call a subprocess and print its output to stdout at realtime . This works just fine if I call the python script from the console. However, when calling it from a jupyter notebook the code hangs at the line proc.stdout.readline(). All previous print statements work fine..

proc = subprocess.Popen(["calc", "input.txt"], cwd=__dir, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
    out = proc.stdout.readline().decode("utf-8")
    err = proc.stderr.readline().decode("utf-8")
    if out == '' and err == '' and proc.poll() is not None:
        break
    if out:
        print("::::%s"%(out), end='')
    if err:
        print("::::%s"%(err), end='', file=sys.stderr)
rc = proc.poll()
print("---> returns %s"%(rc))

Does anyone have an idea on how to fix this?

Check if [this](https://stackoverflow.com/questions/56138384/capture-jupyter-notebook-stdout-with-subprocess) helps. — yorodm, Nov 22 '20 at 15:07

score 1 · Answer 1 · edited Nov 23 '20 at 14:38

I use this custom function to execute a command in Notebooks.

import subprocess

def run_cmd(cmd: str, stderr=subprocess.STDOUT) -> None:
    """Run a command in terminal

    Args:
        cmd (str): command to run in terminal
        stderr (subprocess, optional): Where the error has to go. Defaults to subprocess.STDOUT.

    Raises:
        e: Excetion of the CalledProcessError
    """
    out = None
    try:
        out = subprocess.check_output(
            [cmd],
            shell=True,
            stderr=stderr,
            universal_newlines=True,
        )
    except subprocess.CalledProcessError as e:
        print(f'ERROR {e.returncode}: {cmd}\n\t{e.output}',
              flush=True, file=sys.stderr)
        raise e
    print(out)

Usecase:

run_cmd("pip install emoji")

But will this print realtime output (say if the command takes a long time to run)? My question has already been answered in a different post as pointed out by @yorodm — user2224350, Nov 26 '20 at 22:14

Print realtime output of subprocess in jupyter notebook

1 Answers1