Limited buffer in Popen

Question

I am launching a script using a python code. This code should launch the script which writes a file on the disk, and wait for the script to finish.

But whenever I launch this script using python, the result file doesn't exceed 65768 bytes and the script doesn't respond anymore. Here is what I use in python :

p = subprocess.Popen(command, 
                     shell=True, 
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.STDOUT, 
                     bufsize=-1)
p.wait()

where command is the command for the script.

Does anyone know a solution for this issue ?

Thank you.

Does the script also echoes (something lengthy, such as a copy of what is written into the file) to stdout ? Indeed, you are not reading the child's stdout, so if it becomes full at some point, the child will stop everything (hanging on e.g. a `print` statement), including writing to its output file. — , Feb 25 '14 at 12:56
@OneOfOne: it is not a duplicate. This question is not about `MemoryError`; it is about deadlocking due to the full pipe buffer. — jfs, Feb 25 '14 at 18:15
If you don't need the child process' stdout then you could redirect it to devnull: `import os; DEVNULL=open(os.devnull, 'wb'); subprocess.check_call([script], stdout=DEVNULL, stderr=STDOUT)` — jfs, Feb 25 '14 at 18:19

Torxed · Accepted Answer · 2014-02-25T19:58:32.453

2

from time import sleep
p = subprocess.Popen(command, 
                     shell=True, 
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.STDOUT, 
                     bufsize=-1)
output = ''
while p.poll() is None:
    output += p.stdout.readline()+'\n'  # <--- This is your magic
    sleep(0.025)
output += p.stdout.read() # <--- And this is just to get the leftover data
print('Command finished')

p.stdout.close()

as @J.F comments on almost every single Popen answer i give, you should never define stdout=..., stdin=... or stderr=... unless you're going to utelize them. Because they will fill up the buffer and hang your application.

But if you do, make sure you "tap" from it once in a while.

edited Feb 25 '14 at 19:58

answered Feb 25 '14 at 13:02

Torxed

22,866
14
82
131

Use `output = p.communicate()[0]` instead of the lines starting at `output=''` and to the end. To discard output; you could [use `DEVNULL`](http://stackoverflow.com/questions/22014648/limited-buffer-in-popen#comment33383993_22014648). – jfs Feb 25 '14 at 18:21
2

I remember commenting about why `is None` should be used instead of `== None`. – jfs Feb 25 '14 at 18:22
@J.F.Sebastian Hahaha, ah man.. Old habits and all that.. (Also, no disrespect to your.. I do learn, just some stuff takes a while before it settles in to the daily routines of fast coding) – Torxed Feb 25 '14 at 19:58

Limited buffer in Popen

1 Answers1