How to avoid a Broken Pipe error when printing a large amount of formatted data?

Question

I am trying to print a list of tuples formatted in my stdout. For this, I use the str.format method. Everything works fine, but when I pipe the output to see the first lines using the head command a IOError occurs.

Here is my code:

# creating the data
data = []$
for i in range(0,  1000):                                            
  pid = 'pid%d' % i
  uid = 'uid%d' % i
  pname = 'pname%d' % i
  data.append( (pid, uid, pname) )

# find max leghed string for each field
pids, uids, pnames = zip(*data)
max_pid = len("%s" % max( pids) )
max_uid = len("%s" % max( uids) )
max_pname = len("%s" % max( pnames) )

# my template for the formatted strings
template = "{0:%d}\t{1:%d}\t{2:%d}" % (max_pid, max_uid, max_pname)

# print the formatted output to stdout
for pid, uid, pname in data:
  print template.format(pid, uid, pname)

And here is the error I get after running the command: python myscript.py | head

Traceback (most recent call last):
  File "lala.py", line 16, in <module>
    print template.format(pid, uid, pname)
IOError: [Errno 32] Broken pipe

Can anyone help me on this?

I tried to put print in a try-except block to handle the error, but after that there was another message in the console:

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

I also tried to flush immediately the data through a two consecutive sys.stdout.write and sys.stdout.flush calls, but nothing happend..

This happens because `head` *closes* `stdout`, causing `print` to try and write to a closed file. What would you like to happen instead? — Martijn Pieters, Apr 03 '13 at 17:19
Ok, thank you! I would like to avoid the printing of such messages in the console. I want to use a variation of this code to a command line tool. — Thanasis Petsas, Apr 03 '13 at 17:24
This question is a possible duplicate; see: http://stackoverflow.com/questions/11423225/why-does-my-python3-script-balk-at-piping-its-output-to-head-or-tail-sys-module — , Jul 20 '14 at 12:36
Duplicate of: [IOError: \[Errno 32\] Broken pipe when piping: \`prog.py | othercmd\`](https://stackoverflow.com/questions/14207708/ioerror-errno-32-broken-pipe-when-piping-prog-py-othercmd) — Nickolay, Mar 30 '21 at 14:51

Martijn Pieters · Accepted Answer · 2013-04-03T17:42:18.343

13

head reads from stdout then closes it. This causes print to fail, internally it writes to sys.stdout, now closed.

You can simply catch the IOError and exit silently:

try:
    for pid, uid, pname in data:
        print template.format(pid, uid, pname)
except IOError:
    # stdout is closed, no point in continuing
    # Attempt to close them explicitly to prevent cleanup problems:
    try:
        sys.stdout.close()
    except IOError:
        pass
    try:
        sys.stderr.close()
    except IOError:
        pass

edited Apr 03 '13 at 17:42

answered Apr 03 '13 at 17:26

Martijn Pieters

1,048,767
296
4,058
3,343

ok in this simple program this works, but in the command line tool sometimes works and sometimes occurs again this message: `close failed in file object destructor: sys.excepthook is missing lost sys.stderr` – Thanasis Petsas Apr 03 '13 at 17:34
Haha! I found it too in some answer at stackoverflow! i tried it but it doesn't work! It seems that there is a race condition.. But I cannot figure out how to solve it.. – Thanasis Petsas Apr 03 '13 at 17:46
2

ok I found it! It needs just an addition to your answer. Just put a `sys.stdout.flush()` right after the print statement!! But the error is what you described! I also noticed that for some reason if I put the flush out of the loop it works too.. Anyway. thank you! :) – Thanasis Petsas Apr 03 '13 at 18:04
@ThanasisPetsas: The flush can be affected by the close too. – Martijn Pieters Apr 03 '13 at 18:04
I used the first version of your answer with the `flush` after the `print` and everything works great! – Thanasis Petsas Apr 03 '13 at 18:05
1

Glad to have been of help then. :-) The flush may work for you because when using pipes python uses a regular file buffer instead of a line buffer, and flushing helps get your data out line by line in that case. – Martijn Pieters Apr 03 '13 at 18:06
Thank you very much! I have come across you answers many times and they have helped me! :) – Thanasis Petsas Apr 03 '13 at 22:17
unfortunately this doesn't work in Python 3. Python 3 raises BrokenPipeError, but _io.TextIOWrapper catches it and emits it as a warning. So the failure isn't quite as bad but still ugly and annoying, and there's no way I can see yet to suppress it. – zzzeek Mar 10 '14 at 23:43
Warnings can be silenced with the `warnings` module; or is this not one of those? – Martijn Pieters Mar 10 '14 at 23:56

score 2 · Answer 2 · answered May 17 '16 at 20:18

The behavior you are seeing is linked to the buffered output implementation in Python3. The problem can be avoided using the -u option or setting environmental variable PYTHONUNBUFFERED=x. See the man pages for more information on -u.

$ python2.7 testprint.py | echo

Exc: <type 'exceptions.IOError'>
$ python3.5 testprint.py | echo

Exc: <class 'BrokenPipeError'>
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe
$ python3.5 -u testprint.py | echo

Exc: <class 'BrokenPipeError'>
$ export PYTHONUNBUFFERED=x
$ python3.5 testprint.py | echo

Exc: <class 'BrokenPipeError'>

Yup, but I had this problem in Python2.7. I don't use Python3. — Thanasis Petsas, May 18 '16 at 14:21

score 2 · Answer 3 · answered Mar 07 '17 at 23:47

In general, I try to catch the most specific error I can get away with. In this case it is BrokenPipeError:

try:
    # I usually call a function here that generates all my output:
    for pid, uid, pname in data:
        print template.format(pid, uid, pname)
except BrokenPipeError as e:
    pass  # Ignore. Something like head is truncating output.
finally:
    sys.stderr.close()

If this is at the end of execution, I find I only need to close sys.stderr. If I don't close sys.stderr, I'll get a BrokenPipeError but without a stack trace.

This seems to be the minimum fix for writing tools that output to pipelines.

Gringo Suave · Answer 4 · 2017-10-14T18:58:39.450

Had this problem with Python3 and debug logging piped into head as well. If your script talks to the network or does file IO, simply dropping IOError's is not a good solution. Despite mentions here, I was not able to catch BrokenPipeError for some reason.

Found a blog post talking about restoring the default signal handler for sigpipe: http://newbebweb.blogspot.com/2012/02/python-head-ioerror-errno-32-broken.html

In short, you add the following to your script before the bulk of the output:

if log.isEnabledFor(logging.DEBUG):  # optional
    # set default handler to no-op
    from signal import signal, SIGPIPE, SIG_DFL
    signal(SIGPIPE, SIG_DFL)

This seems to happen with head, but not other programs such as grep---as mentioned head closes stdout. If you don't use head with the script often, it may not be worth worrying about.

Great! Thanks for the explanation and the article! – Thanasis Petsas Oct 16 '17 at 12:12 — Thanasis Petsas, Oct 16 '17 at 12:12

How to avoid a Broken Pipe error when printing a large amount of formatted data?

4 Answers4

Linked