How to handle a broken pipe (SIGPIPE) in python?

Question

I've written a simple multi-threaded game server in python that creates a new thread for each client connection. I'm finding that every now and then, the server will crash because of a broken-pipe/SIGPIPE error. I'm pretty sure it is happening when the program tries to send a response back to a client that is no longer present.

What is a good way to deal with this? My preferred resolution would simply close the server-side connection to the client and move on, rather than exit the entire program.

PS: This question/answer deals with the problem in a generic way; how specifically should I solve it?

mhawke · Answer 1 · 2008-10-08T02:23:50.767

Assuming that you are using the standard socket module, you should be catching the socket.error: (32, 'Broken pipe') exception (not IOError as others have suggested). This will be raised in the case that you've described, i.e. sending/writing to a socket for which the remote side has disconnected.

import socket, errno, time

# setup socket to listen for incoming connections
s = socket.socket()
s.bind(('localhost', 1234))
s.listen(1)
remote, address = s.accept()

print "Got connection from: ", address

while 1:
    try:
        remote.send("message to peer\n")
        time.sleep(1)
    except socket.error, e:
        if isinstance(e.args, tuple):
            print "errno is %d" % e[0]
            if e[0] == errno.EPIPE:
               # remote peer disconnected
               print "Detected remote disconnect"
            else:
               # determine and handle different error
               pass
        else:
            print "socket error ", e
        remote.close()
        break
    except IOError, e:
        # Hmmm, Can IOError actually be raised by the socket module?
        print "Got IOError: ", e
        break

Note that this exception will not always be raised on the first write to a closed socket - more usually the second write (unless the number of bytes written in the first write is larger than the socket's buffer size). You need to keep this in mind in case your application thinks that the remote end received the data from the first write when it may have already disconnected.

You can reduce the incidence (but not entirely eliminate) of this by using select.select() (or poll). Check for data ready to read from the peer before attempting a write. If select reports that there is data available to read from the peer socket, read it using socket.recv(). If this returns an empty string, the remote peer has closed the connection. Because there is still a race condition here, you'll still need to catch and handle the exception.

Twisted is great for this sort of thing, however, it sounds like you've already written a fair bit of code.

This looks strange `if isinstance(e.args, tuple):`. Can someone explain this? — guettli, May 12 '16 at 14:37
Why is EPIPE not always raised on first write? What is the condition that EPIPE is raised? — tartaruga_casco_mole, Jan 22 '21 at 16:05

score 41 · Accepted Answer · edited Jun 14 '11 at 20:45

41

Read up on the try: statement.

try:
    # do something
except socket.error, e:
    # A socket error
except IOError, e:
    if e.errno == errno.EPIPE:
        # EPIPE error
    else:
        # Other error

edited Jun 14 '11 at 20:45

John Zwinck

239,568
38
324
436

answered Oct 07 '08 at 20:07

S.Lott

384,516
81
508
779

If I do a try: #something except: #anything, will it just catch anything, and not just IOErrors? – Adam Plumb Oct 07 '08 at 20:28
6

The blanket except is a bad policy. But yet, it will catch any kind of exception. You know it's an IOError. Handle that. If something else crops up, figure out why and handle it appropriately. You don't want to mask bugs like division by zero or out of memory. – S.Lott Oct 07 '08 at 20:46
1

If you are using Python's socket module, you will not get an IOError exception: you will get a socket.error exception. – mhawke Oct 08 '08 at 00:17
You are not going to get an IOError. socket.error does not have the errno attribute for broken pipe - this code will raise AttributeError. – mhawke Oct 08 '08 at 00:55
10

You won't get IOError with errno == EPIPE for broken pipe socket exceptions, you will get socket.error, so there's no point in checking for it in the IOError exception handler. You have 2 votes for a (still) bad answer. Perhaps you should up vote my answer :) – mhawke Oct 08 '08 at 01:27
At this point, the questioner should know how to use the try: statement. – S.Lott Oct 08 '08 at 01:39
2

Agree that the questioner should now have some clue as to what to do. You code snippet is still incorrect though, and the questioner might follow your example. It won't break his code, it's just that checking for EPIPE in the IOError handler is useless. – mhawke Oct 08 '08 at 01:45
2

@mhawke: you're still right. Both times. However, it's hard to contrive an example of standard OS errors (with errno) and other errors (without errno). I think it's important to have a tidy code sample -- I'm not writing their app for them. – S.Lott Oct 08 '08 at 01:54
Blanket excepts may usually be a bad policy but in this case it is really not. This is more or less what exceptions are for. It is almost always better to use if statements to test that your given inputs and calls won't throw errors, because that won't screw up your debugging. However sockets have lower level resources then can be easily accessed with the python socket module. Since we can't really access the pipe and handle it appropriately on the lower level, exceptions are the way to go. – trevorKirkby Nov 07 '14 at 17:37

score 3 · Answer 3 · answered Oct 07 '08 at 20:55

SIGPIPE (although I think maybe you mean EPIPE?) occurs on sockets when you shut down a socket and then send data to it. The simple solution is not to shut the socket down before trying to send it data. This can also happen on pipes, but it doesn't sound like that's what you're experiencing, since it's a network server.

You can also just apply the band-aid of catching the exception in some top-level handler in each thread.

Of course, if you used Twisted rather than spawning a new thread for each client connection, you probably wouldn't have this problem. It's really hard (maybe impossible, depending on your application) to get the ordering of close and write operations correct if multiple threads are dealing with the same I/O channel.

*The simple solution is not to shut the socket down before trying to send it data.* You assume here that the socket was shut down locally (on the server side) whereas in [this](http://stackoverflow.com/a/11866962/95735) answer we read that *This usually happens when you write to a socket fully closed on the other (client) side.* Did you omit this case by purpose or you don't agree with this statement? — Piotr Dobrogost, Dec 02 '14 at 15:35

score -1 · Answer 4 · answered Oct 26 '17 at 05:46

I face with the same question. But I submit the same code the next time, it just works. The first time it broke:

$ packet_write_wait: Connection to 10.. port 22: Broken pipe

The second time it works:

[1]   Done                    nohup python -u add_asc_dec.py > add2.log 2>&1

I guess the reason may be about the current server environment.

score -3 · Answer 5 · answered Oct 07 '08 at 21:05

-3

My answer is very close to S.Lott's, except I'd be even more particular:

try:
    # do something
except IOError, e:
    # ooops, check the attributes of e to see precisely what happened.
    if e.errno != 23:
        # I don't know how to handle this
        raise

where "23" is the error number you get from EPIPE. This way you won't attempt to handle a permissions error or anything else you're not equipped for.

answered Oct 07 '08 at 21:05

Kirk Strauser

30,189
5
49
65

2

The errno would be 32, not 23. – mhawke Oct 08 '08 at 00:24
2

I should have clarified that I meant "23" as a placeholder. Really? 32? I was closer than I would have guessed. :-) – Kirk Strauser Oct 08 '08 at 02:55

How to handle a broken pipe (SIGPIPE) in python?

5 Answers5

Linked

Related