python 3 non blocking p.stdout.readline()

Question

Hi guys I am trying to read from subprocess.PIPE without blocking the main process. I have found this code:

import sys
from subprocess import PIPE, Popen
from threading  import Thread

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # python 3.x

ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()

p = Popen(['myprogram.exe'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()

# ... do other things here

# read line without blocking
try:  line = q.get_nowait() # or q.get(timeout=.1)
except Empty:
    print('no output yet')
else: # got line
    # ... do something with line

The code does not return anything. I am using Python 3 on Windows. Do you have any ideas what might be the problem?

[It's somewhere in this presentation, probably close to the front.](https://www.youtube.com/watch?v=rrMnmLyYjU8) — ddg, Jan 30 '18 at 01:12
I have watched the presentation but it is still not quite clear for me where the problem is. I am new to Python and the way I learn it, is by looking at codes. I have to read more about threads I suppose. — azzza, Feb 01 '18 at 23:16
The origin of the code: [A non-blocking read on a subprocess.PIPE in Python](https://stackoverflow.com/q/375427/4279) — jfs, May 06 '23 at 05:22

score 1 · Answer 1 · answered Apr 17 '18 at 02:24

I was away from my project for a long time but finally I manged to solve the issue.

from subprocess import PIPE, Popen
from threading  import Thread

p = Popen(['myprogram.exe'], stdout=PIPE)
t = Thread(target=results)
t.daemon = True
t.start()


def results():
    a = p.stdout.readline()

Maybe this is not exactly the right way to do it but it is working for me. I am only posting it because I personally believe that whoever ask a question should post the solution when they have found it.

Rune Kaagaard · Answer 2 · 2022-04-07T22:15:50.043

On a unix environment you can simply make the stdout/stderr/stdin file descriptors nonblocking like so:

import os, fcntl
from subprocess import Popen, PIPE

def nonblock(stream):
    fcntl.fcntl(stream, fcntl.F_SETFL, fcntl.fcntl(stream, fcntl.F_GETFL) | os.O_NONBLOCK)

proc = Popen("for ((;;)) { date; sleep 1; }", shell=True, stdout=PIPE, stderr=PIPE, universal_newlines=True,
             executable='/bin/bash')
nonblock(proc.stdout)

while True:
    for line in proc.stdout.readlines():
        print(line, end="")

python 3 non blocking p.stdout.readline()

2 Answers2