gevent tcp socket is disconnected but can send data

Question

I have a small HTTP server written using gevent that accepts incoming connections and sends back data using a per-connection greenlet. When the (local) client dies server-side writes to the client succeed despite the socket being in the CLOSE_WAIT state:

$ ls -l /proc/21860/fd |grep 22
lrwx------. 1 mathieu mathieu 64 Mar 21 19:55 22 -> socket:[187093]
$ lsof |grep 187093
python    21860       mathieu   22u     IPv4             187093       0t0        TCP localhost.localdomain:36072->localhost.localdomain:48908 (CLOSE_WAIT)

Now, I would expect the send method of the socket I created with gevent.socket.socket to fail with an exception but I get none !

def send(socket, data):
    sent = socket.send(data)
    while sent != len(data):
        data = data[sent:]
        sent = socket.send(data)

Is my expectation incorrect ? If not, what could be going wrong here ?

Answer 1

After a bit of debugging, it turns out that my expectation about how the socket API behaves on top of TCP was incorrect. Potential readers might want to read carefully TCP option SO_LINGER (zero) - when it's required

Things to take away:

1. client-side close() sends FIN to the server which roughly means "I am done sending you data". This means that the server might still be able to send back data.
2. If the server attempts a write after a client-side close, the write will succeed (returns number of bytes copied from userspace to kernelspace), the data will be sent to the client, the client will send back RST because its socket is still in TIMEWAIT state (unless you disabled this with SO_LINGER zero which is why you really do not want to set SO_LINGER to zero), the server socket state will be updated to be CLOSED.
3. If the server attempts a second write after the RST has been received from the client, the write will fail.
4. If the server attempts a recv after a client-side close, the recv will return zero to indicate end-of-stream.

To sum it up, it means that my server needs to use the end-of-stream notification returned as recv() == 0 as a sign that the client has closed its connection and act accordingly (that is, close the associated socket).