Stuck in infinite loop while trying to read all lines in proc.stdout.readline

Question

I am trying to read each line in proc.stdout.readline and send the lines over the network, for example:

data = b''
for line in iter(proc.stdout.readline, ''):
    data += line
clientsocket.send(data)

When I run this code I seem to be stuck in a inifinite loop unable to escape to the line:

clientsocket.send(data)

Is there a more efficient way to read the data? I've tried also with a while loop and breaking 'if not line':

data = b''
while True:
    line += proc.stdout.readline()
    data += line
    if not line:
        break
clientsocket.send(data)

This seems to also produce the same results. Is there a more efficient way to read all of the data from proc.stdout.readline?

Ok, so its not stuck in an infinite loop, thing is that the `proc.stdout.readline` never receives a `EOF` , so it is waiting to read more characters. — Rahul Bharadwaj, Jan 17 '20 at 13:39
``data`` suggests that ``stdout`` produces ``bytes``. The sentinel of ``''`` is a ``str``, which will never equal anything ``stdout`` produces due to mismatching types. — MisterMiyagi, Jan 17 '20 at 13:47

score 1 · Answer 1 · answered Jul 04 '20 at 10:35

I've encountered the same very problem. The strange thing that in Python 2.7 it had no problem to converge and actually stop iterating. During debug (in Python 3.5) I've noticed that all true lines returned with the '\n' character, whereas the line that wasn't suppose to arrive returned as an empty string, i.e. ''. So, I just added an if-clause checking against '' and breaking the loop if positive.
My final version looks as follows:

lines = []
for _line in iter(process.stdout.readline, b''):
    if _line == '':
        break
    lines.append(_line)

One thing that might be worth to mention, is that I used universal_newlines=True argument upon subprocess.Popen(..) call.

score 0 · Answer 2 · answered Jan 17 '20 at 13:45

0

The statement: iter(proc.stdout.readline, "") will do a blocking read until it recieves an EOF.

If you want to read all the lines, then you can just do:

data = b''
data = b"".join(proc.stdout.readlines())

There is no other solution than for the proc to produce lines faster. If you want, you can read lines with timeout (i.e. you can wait to read a select number of characters, or timeout if that number of characters are not read).

Those answers can be found here:

answered Jan 17 '20 at 13:45

Rahul Bharadwaj

2,555
2
18
29

This doesn't seem to be working either, perhaps I should resort to a timeout then? – Alfred Relory Jan 17 '20 at 13:53
Are you sure that the subprocess returns some output? – Rahul Bharadwaj Jan 17 '20 at 13:59
I am positive it returns output – Alfred Relory Jan 17 '20 at 14:03
I see, if the subprocess is taking lot of time to return full output, then you can return partial outputs by using timeouts. Apart from that, I can see no other solution. – Rahul Bharadwaj Jan 17 '20 at 14:04
The clue is to use `os.read` after `select.poll`, instead of `proc.stdout.readlines`. – Matt Shin Jan 17 '20 at 14:42

Stuck in infinite loop while trying to read all lines in proc.stdout.readline

2 Answers2