How to recover from network interruption using boost::asio

Question

I am writing a server that accepts data from a device and processes it. Everything works fine unless there is an interruption in the network (i.e., if I unplug the Ethernet cable, then reconnect it). I'm using read_until() because the protocol that the device uses terminates the packet with a specific sequence of bytes. When the data stream is interrupted, read_until() blocks, as expected. However when the stream starts up again, it remains blocked. If I look at the data stream with Wireshark, the device continues transmitting and each packet is being ACK'ed by the network stack. But if I look at bytes_readable it is always 0. How can I detect the interruption and how to re-establish a connection to the data stream? Below is a code snippet and thanks in advance for any help you can offer. [Go easy on me, this is my first Stack Overflow question....and yes I did try to search for an answer.]

using boost::asio::ip::tcp;

boost::asio::io_service IOservice;
tcp::acceptor acceptor(IOservice, tcp::endpoint(tcp::v4(), listenPort));
tcp::socket socket(IOservice);
acceptor.accept(socket);

for (;;)
{
    len = boost::asio::read_until(socket, sbuf, end);
    // Process sbuf
    // etc.
}

Tom Trebicky · Answer 1 · 2017-08-18T18:36:48.617

Remember, the client initiates a connection, so the only thing you need to achieve is to re-create the socket and start accepting again. I will keep the format of your snippet but I hope your real code is properly encapsulated.

using SocketType = boost::asio::ip::tcp::socket;

std::unique_ptr<SocketType> CreateSocketAndAccept(
    boost::asio::io_service& io_service,
    boost::asio::ip::tcp::acceptor& acceptor) {
  auto socket = std::make_unique<boost::asio::ip::tcp::socket>(io_service);
  boost::system::error_code ec;
  acceptor.accept(*socket.get(), ec);
  if (ec) {
    //TODO: Add handler.
  }
  return socket;
}

...
auto socket = CreateSocketAndAccept(IOservice, acceptor);
for (;;) {
  boost::system::error_code ec;
  auto len = boost::asio::read_until(*socket.get(), sbuf, end, ec);
  if (ec)  // you could be more picky here of course,
           // e.g. check against connection_reset, connection_aborted
    socket = CreateSocketAndAccept(IOservice, acceptor);
  ...
}

Footnote: Should go without saying, socket needs to stay in scope.

Edit: Based on the comments bellow.

The listening socket itself does not know whether a client is silent or whether it got cut off. All operations, especially synchronous, should impose a time limit on completion. Consider setting SO_RCVTIMEO or SO_KEEPALIVE (per socket, or system wide, for more info How to use SO_KEEPALIVE option properly to detect that the client at the other end is down?).

Another option is to go async and implement a full fledged "shared" socket server (BOOST example page is a great start).

Either way, you might run into data consistency issues and be forced to deal with it, e.g. when the client detects an interrupted connection, it would resend the data. (or something more complex using higher level protocols)

Thanks for that Tom. However the problem is that read_until() doesn't return an error code when I pull the plug. It simply blocks and never returns. — Schnizz, Aug 18 '17 at 17:17
Thinking about it now, I want to say it makes sense. The server (receiving socket) has no way of differentiating between inactivity on the wire and the ungracefully shutdown connection. I would consider either setting `SO_RCVTIMEO` on the socket or implementing an asynchronous wrapper around a deadline timer to stop the io_service running the asynchronous read. Either way, I would say you need an external piece of information on the timeout. That said, I would recommend going full async with the ability to create multiple "shared" sockets for incoming, accepted TCP connections. — Tom Trebicky, Aug 18 '17 at 17:52
Yeah I was wondering whether full async would solve the problem. I'll give that a go. Appreciate the help. — Schnizz, Aug 18 '17 at 18:19

score 1 · Answer 2 · answered Aug 18 '17 at 18:47

If you want to stay synchronous, the way I've seen things handled is to destroy the socket when you detect an interruption. The blocking call should throw an exception that you can catch and then start accepting connections again.

for (;;)
{
    try {
        len = boost::asio::read_until(socket, sbuf, end);
        // Process sbuf
        // etc.
    }
    catch (const boost::system::system_error& e) {
     // clean up. Start accepting new connections.
    }
}

As Tom mentions in his answer, there is no difference between inactivity and ungraceful disconnection so you need an external mechanism to detect this.

If you're expecting continuous data transfer, maybe a timeout per connection on the server side is enough. A simple ping could also work. After accepting a connection, ping your client every X seconds and declare the connection dead if he doesn't answer.

It's worth mentioning that exceptions are equivalent to using `boost::system::error_code`. In fact, the former is implemented (thrown) on top of an ordinary syscall error. — Tom Trebicky, Aug 18 '17 at 19:41
I implemented a timeout as Tom suggested and that works in Windows but not in Linux. However long term, async is probably a more graceful option. So at some point I'll refactor it to use async methods. Thanks both of you for your help. — Schnizz, Aug 18 '17 at 21:39

How to recover from network interruption using boost::asio

2 Answers2