17

First, a little background to explain the motivation: I'm working on a very simple select()-based TCP "mirror proxy", that allows two firewalled clients to talk to each other indirectly. Both clients connect to this server, and as soon as both clients are connected, any TCP bytes sent to the server by client A is forwarded to client B, and vice-versa.

This more or less works, with one slight gotcha: if client A connects to the server and starts sending data before client B has connected, the server doesn't have anywhere to put the data. I don't want to buffer it up in RAM, since that could end up using a lot of RAM; and I don't want to just drop the data either, as client B might need it. So I go for the third option, which is to not select()-for-read-ready on client A's socket until client B has also connected. That way client A just blocks until everything is ready to go.

That more or less works too, but the side effect of not selecting-for-read-ready on client A's socket is that if client A decides to close his TCP connection to the server, the server doesn't get notified about that fact -- at least, not until client B comes along and the server finally selects-for-read-ready on client A's socket, reads any pending data, and then gets the socket-closed notification (i.e. recv() returning 0).

I'd prefer it if the server had some way of knowing (in a timely manner) when client A closed his TCP connection. Is there a way to know this? Polling would be acceptable in this case (e.g. I could have select() wake up once a minute and call IsSocketStillConnected(sock) on all sockets, if such a function existed).

Jeremy Friesner
  • 70,199
  • 15
  • 131
  • 234
  • You may want to consider [this question](http://stackoverflow.com/q/16320039/315052). – jxh Jul 17 '13 at 18:01

7 Answers7

10

If you want to check if the socket is actually closed instead of data, you can add the MSG_PEEK flag on recv() to see if data arrived or if you get 0 or an error.

/* handle readable on A */
if (B_is_not_connected) {
    char c;
    ssize_t x = recv(A_sock, &c, 1, MSG_PEEK);
    if (x > 0) {
        /* ...have data, leave it in socket buffer until B connects */
    } else if (x == 0) {
        /* ...handle FIN from A */
    } else {
        /* ...handle errors */
    }
}

Even if A closes after sending some data, your proxy probably wants to forward that data to B first before forwarding the FIN to B, so there is no point in knowing that A has sent FIN on the connection sooner than after having read all the data it has sent.

A TCP connection isn't considered closed until after both sides send FIN. However, if A has forcibly shutdown its endpoint, you will not know that until after you attempt to send data on it, and receive an EPIPE (assuming you have suppressed SIGPIPE).

After reading your mirror proxy application a bit more, since this is a firewall traversal application, it seems that you actually need a small control protocol to allow to you verify that these peers are actually allowed to talk to each other. If you have a control protocol, then you have many solutions available to you, but the one I would advocate would be to have one of the connections describe itself as the server, and the other connection describe itself as the client. Then, you can reset the connection the client if there is no server present to take its connection. You can let servers wait for a client connection up to some timeout. A server should not initiate any data, and if it does without a connected client, you can reset the server connection. This eliminates the issue of buffering data for a dead connection.

jxh
  • 69,070
  • 8
  • 110
  • 193
  • If I leave A's socket readable, but don't actually read the data, then select() will always return immediately and my server process will chew up 100% of a CPU core, no? – Jeremy Friesner Jul 17 '13 at 17:24
  • Yeah, forgot that `select` is level triggered. You still want to leave it readable for the first read to check if all you got was a FIN, though, which I thought was what you asked. Then you can remove it after this check if B is still not connected. I usually use edge triggered semantics with `epoll` on linux. – jxh Jul 17 '13 at 17:28
  • I see. I suspect that the recv(MSG_PEEK) call wouldn't indicate a FIN until after I had read any and all pending TCP data from A's socket, though... at which point I'm back to the original problem (what to do with all this data that I've read from A's socket, with no B to send it to) – Jeremy Friesner Jul 17 '13 at 17:32
  • The sender will expect that all the data in front of its FIN will get delivered to the destination. So the right thing to do is to forward the data, then forward the FIN. You don't mention what protocol this is, or if you have the power to change the protocol. If you can change the protocol, you can allow the proxy to pretend to be B that is only sending innocuous messages (something similar to an application level PING). This way, A will just discard the messages if the connection is still up, or send RST if the connection is actually down. If you get RST, you can tear down your connection. – jxh Jul 17 '13 at 17:40
  • Well, maybe, but if client B doesn't connect to the server until hours/days/weeks after client A gave up and closed his socket, B may end up getting a short burst of hopelessly out-of-date TCP data from A's leftover "zombie connection" (which then closes immediately afterwards) -- behavior which doesn't seem very useful for my purposes. – Jeremy Friesner Jul 17 '13 at 17:40
  • Ideally the mirror proxy would be protocol-neutral and could work for any TCP protocol; perhaps that is too ambitious though :) – Jeremy Friesner Jul 17 '13 at 17:41
  • You don't really explain what the purpose of the proxy is, so I can't really advise then. If it were me, I probably would just defer allowing A to connect unless there was a B. But I say this without knowing what purpose your proxy is serving, so I don't know if the suggestion is valid. – jxh Jul 17 '13 at 17:44
2

It appears the answer to my question is "no, not unless you are willing and able to modify your TCP stack to get access to the necessary private socket-state information".

Since I'm not able to do that, my solution was to redesign the proxy server to always read data from all clients, and throw away any data that arrives from a client whose partner hasn't connected yet. This is non-optimal, since it means that the TCP streams going through the proxy no longer have the stream-like property of reliable in-order delivery that TCP-using programs expect, but it will suffice for my purpose.

Jeremy Friesner
  • 70,199
  • 15
  • 131
  • 234
2

For me the solution was to poll the socket status.

On Windows 10, the following code seemed to work (but equivalent implementations seem to exist for other systems):

WSAPOLLFD polledSocket;
polledSocket.fd = socketItf;
polledSocket.events = POLLRDNORM | POLLWRNORM;

if (WSAPoll(&polledSocket, 1, 0) > 0)
{
    if (polledSocket.revents &= (POLLERR | POLLHUP))
    {
        // socket closed
        return FALSE;
    }
}
Random Profile
  • 21
  • 3
2

Since Linux 2.6.17, you can poll/epoll for POLLRDHUP/EPOLLRDHUP. See epoll_ctl(2):

EPOLLRDHUP (since Linux 2.6.17)

Stream socket peer closed connection, or shut down writing half of connection. (This flag is especially useful for writing simple code to detect peer shutdown when using Edge Triggered monitoring.)

Ian Chen
  • 23
  • 5
1

I don't see the problem as you see it. Let's say A connects to the server sends some data and close, it does not need any message back. Server won't read its data until B connects, once it does server read socket A and send the data to B. The first read will return the data A had sent and the second return either 0 or -1 in either case the socket is closed, server close B. Let's suppose A send a big chunk of data, the A's send() method will block until server starts reading and consumes the buffer.

I would use a function with a select which returns 0, 1, 2, 11, 22 or -1, where;

  • 0=No data in either socket (timeout)
  • 1=A has data to read
  • 2=B has data to read
  • 11=A socket has an error (disconnected)
  • 22=B socket has an error (disconnected)

  • -1: One/both socket is/are not valid

    int WhichSocket(int sd1, int sd2, int seconds, int microsecs) { 
   fd_set sfds, efds;
   struct timeval timeout={0, 0};
   int bigger;
   int ret;


   FD_ZERO(&sfds);
   FD_SET(sd1, &sfds);
   FD_SET(sd2, &sfds);
   FD_SET(sd1, &efds);
   FD_SET(sd2, &efds);
   timeout.tv_sec=seconds;
   timeout.tv_usec=microsecs;
   if (sd1 > sd2) bigger=sd1;
   else bigger=sd2;
   // bigger is necessary to be Berkeley compatible, Microsoft ignore this param.
   ret = select(bigger+1, &sfds, NULL, &efds, &timeout);
   if (ret > 0) {
       if (FD_ISSET(sd1, &sfds)) return(1); // sd1 has data
       if (FD_ISSET(sd2, &sfds)) return(2); // sd2 has data
       if (FD_ISSET(sd1, &efds)) return(11);    // sd1 has an error
       if (FD_ISSET(sd2, &efds)) return(22);    // sd2 has an error
   }
   else if (ret < 0) return -1; // one of the socket is not valid
   return(0); // timeout
}
idmean
  • 14,540
  • 9
  • 54
  • 83
ja_mesa
  • 1,971
  • 1
  • 11
  • 7
0

If your proxy must be a general purpose proxy for any protocol, then you should handle also those clients which sends data and immediately calls close after the send (one way data transfer only).

So if client A sends a data and closes the connection before the connection is opened to B, don't worry, just forward the data to B normally (when connection to B is opened).

There is no need to implement special handling for this scenario.

Your proxy will detect the closed connection when:

  • read returns zero after connection to B is opened and all pending data from A is read. or
  • your programs try to send data (from B) to A.
SKi
  • 8,007
  • 2
  • 26
  • 57
  • A potential problem case would be the one where B never connects at all; then A's connection is held open indefinitely. – Jeremy Friesner Jul 18 '13 at 17:26
  • @Jeremy Friesner : Oh, B is also client. Sorry my mistake. I assumed that the proxy is client for B. – SKi Jul 18 '13 at 19:38
-1

You could check if the socket is still connected by trying to write to the file descriptor for each socket. Then if the return value of the write is -1 or if errno = EPIPE, you know that socket has been closed.
for example

int isSockStillConnected(int *fileDescriptors, int numFDs){
     int i,n;
     for (i=0;i<numFDs;i++){
          n = write(fileDescriptors+i,"heartbeat",9);
          if (n < 0) return -1;
          if (errno == EPIPE) return -1;
     }
     //made it here, must be okay
     return 0;
}
Optox
  • 630
  • 4
  • 9
  • I think the problem with this is that the proxy needs to be transparent -- writing arbitrary data like "heartbeat" to the sockets would confuse the clients, who aren't expecting that. – Jeremy Friesner Jul 17 '13 at 17:25
  • You don't have any sort of check to make sure its correctly formatted data? Alternatively, format it in a way that your clients can handle, but in a way that does nothing. shouldn't be hard to add in on the client side – Optox Jul 17 '13 at 17:39
  • The idea is that (ideally) any program that connects via TCP to a specified hostname/port can use the proxy, without knowing it is connecting to a proxy rather than directly to its peer. – Jeremy Friesner Jul 17 '13 at 17:44