105

If you have a situation where a TCP connection is potentially too slow and a UDP 'connection' is potentially too unreliable what do you use? There are various standard reliable UDP protocols out there, what experiences do you have with them?

Please discuss one protocol per reply and if someone else has already mentioned the one you use then consider voting them up and using a comment to elaborate if required.

I'm interested in the various options here, of which TCP is at one end of the scale and UDP is at the other. Various reliable UDP options are available and each brings some elements of TCP to UDP.

I know that often TCP is the correct choice but having a list of the alternatives is often useful in helping one come to that conclusion. Things like Enet, RUDP, etc that are built on UDP have various pros and cons, have you used them, what are your experiences?

For the avoidance of doubt there is no more information, this is a hypothetical question and one that I hoped would elicit a list of responses that detailed the various options and alternatives available to someone who needs to make a decision.

Len Holgate
  • 21,282
  • 4
  • 45
  • 92
  • 2
    This question appears to be off-topic because it is polling for technologies – Dave Hillier Sep 23 '14 at 19:00
  • Those who thinks TCP is best in all cases, please read: http://en.wikipedia.org/wiki/Bandwidth-delay_product – nullptr Nov 14 '14 at 14:17
  • Wikipedia has a nice [table comparing various aspects of UDP, UDP Lite, TCP, Multipath TCP, SCTP, DCCP, and RUDP](http://en.wikipedia.org/wiki/Transport_layer#Comparison_of_transport_layer_protocols). SCTP supports the most features in that list. – Evgeniy Berezovsky Feb 26 '15 at 05:16
  • @EugeneBeresovsky I made a little research regarding SCTP,most of the info,including from SO answers, date to 2013 and earlier.Most people wrote back then that SCTP adoption was very low.I wonder how is it today?Also,see this thread https://stackoverflow.com/questions/1171555/why-is-sctp-not-much-used-known – Michael IV Jul 24 '17 at 18:19
  • @MichaelIvanov Adoption is low indeed. But if you intend to use it inside your data center, you don't care about outside adoption, as long as switches and routers don't cause problems (which, in a data center, they shouldn't), and you have OS and library support, which may be an issue, as described in [one](https://stackoverflow.com/a/20290710/709537) of the answer in the question you linked to. – Evgeniy Berezovsky Jul 24 '17 at 22:06
  • 1
    It's been a long time since the original question came up and no answer fits what I'm looking for: a layer built on top of UDP to _increase_ reliability of UDP given a known packet loss rate, rather than _guarantee_ reliable delivery. So, no acknowledgements, a configurable share of network capacity set aside to support reliability and that's it. Anyone know of anything like that in the FOSS world? – Tomislav Nakic-Alfirevic Nov 08 '22 at 15:23

15 Answers15

27

What about SCTP. It's a standard protocol by the IETF (RFC 4960)

It has chunking capability which could help for speed.

Update: a comparison between TCP and SCTP shows that the performances are comparable unless two interfaces can be used.

Update: a nice introductory article.

philant
  • 34,748
  • 11
  • 69
  • 112
  • That's good, I'm more interested in things that can be built on top of UDP rather than built on top of IP but it's certainly something that fits in the solution space. – Len Holgate Sep 20 '08 at 10:23
  • SCTP has many great features (such as multihoming) and with the partial reliability extension (RFC 3758) it's an incredibly flexible option. It's included in the latest linux kernel versions, but for windows you'll have to install your own SCTP stack. – Andrew Johnson Oct 05 '08 at 20:10
  • 6
    SCTP can be tunneled over UDP. http://tools.ietf.org/id/draft-ietf-sigtran-sctptunnel-00.txt – Miles Jun 15 '09 at 14:36
  • @Len Holgate UDP is part of the Internet Protocol Suite, so if it's built on top of UDP it is also built on top of IP :) – CrackerJack9 Aug 22 '11 at 18:03
  • 1
    Yes... But something that is built on top of UDP rather than at the same level as UDP is likely to be easier to implement in user space, at least on Windows... – Len Holgate Aug 22 '11 at 20:38
  • I'm well on the way to having implemented a user space version of SCTP over UDP for a WebRTC project that I'm working on. It's nice and, IMHO, better and more flexible than the ENet stuff we were previously using. – Len Holgate Oct 28 '17 at 21:17
26

It's difficult to answer this question without some additional information on the domain of the problem. For example, what volume of data are you using? How often? What is the nature of the data? (eg. is it unique, one off data? Or is it a stream of sample data? etc.) What platform are you developing for? (eg. desktop/server/embedded) To determine what you mean by "too slow", what network medium are you using?

But in (very!) general terms I think you're going to have to try really hard to beat tcp for speed, unless you can make some hard assumptions about the data that you're trying to send.

For example, if the data that you're trying to send is such that you can tolerate the loss of a single packet (eg. regularly sampled data where the sampling rate is many times higher than the bandwidth of the signal) then you can probably sacrifice some reliability of transmission by ensuring that you can detect data corruption (eg. through the use of a good crc)

But if you cannot tolerate the loss of a single packet, then you're going to have to start introducing the types of techniques for reliability that tcp already has. And, without putting in a reasonable amount of work, you may find that you're starting to build those elements into a user-space solution with all of the inherent speed issues to go with it.

Andrew Edgecombe
  • 39,594
  • 3
  • 35
  • 61
  • 5
    Ok, I'll adjust the question. I'm more interested in the pros and cons of the various reliable UDP protocols out there rather than a 'use TCP' response ;) – Len Holgate Sep 20 '08 at 09:22
  • 13
    @Andrew - it's very EASY to beat TCP in two cases: (1) your application has lighter reliability requirements than "all data, always in order, no duplicates, no excessive queueing". Or (2) you are using multicast. Reliable UDP is very common for multicast environments. – Tom Dec 14 '08 at 17:29
  • 4
    Also, TCP suffers horribly when used across a WAN connection (long haul issues). Why, simple. TCP uses windows where the packets in the window have to be ack'd. ACK protocols suffer because of latency due to line distance. Google: WAN TCP "speed of light" – Ajaxx Feb 27 '09 at 03:41
  • 3
    @Ajaxx, you are very correct around this, however, TCP/IP does this purposely because of the last internet meltdown. If you are doing high bit rate protocol without any congestion control, well basically shame on you. If you own the network, then go wild. – Kevin Nisbet Jul 11 '09 at 04:42
  • 2
    "where the sampling rate is significantly higher than the nyquist rate" -- the sampling rate is always twice the nyquist rate, by definition. – Steve Sep 05 '12 at 15:47
  • 1
    @Kevin - some disagree to this being a "shame" on developers/protocols. The so called standard places the burden on consumers to control congestion rather than ISPs. This is like shutting down traffic lights in a major city and leaving the drivers to control congestion - just to save money. There's a second option to dropping packets... add more routers. – JSON Oct 23 '14 at 04:45
25

ENET - http://enet.bespin.org/

I've worked with ENET as a reliable UDP protocol and written an asynchronous sockets friendly version for a client of mine who is using it in their servers. It works quite nicely but I don't like the overhead that the peer to peer ping adds to otherwise idle connections; when you have lots of connections pinging all of them regularly is a lot of busy work.

ENET gives you the option to send multiple 'channels' of data and for the data sent to be unreliable, reliable or sequenced. It also includes the aforementioned peer to peer ping which acts as a keep alive.

Len Holgate
  • 21,282
  • 4
  • 45
  • 92
15

We have some defense industry customers that use UDT (UDP-based Data Transfer) (see http://udt.sourceforge.net/) and are very happy with it. I see that is has a friendly BSD license as well.

Chris Markle
  • 2,076
  • 4
  • 25
  • 46
12

Anyone who decides that the list above isn't enough and that they want to develop their OWN reliable UDP should definitely take a look at the Google QUIC spec as this covers lots of complicated corner cases and potential denial of service attacks. I haven't played with an implementation of this yet, and you may not want or need everything that it provides, but the document is well worth reading before embarking on a new "reliable" UDP design.

A good jumping off point for QUIC is here, over at the Chromium Blog.

The current QUIC design document can be found here.

Len Holgate
  • 21,282
  • 4
  • 45
  • 92
10

RUDP - Reliable User Datagram Protocol

This provides:

  • Acknowledgment of received packets
  • Windowing and congestion control
  • Retransmission of lost packets
  • Overbuffering (Faster than real-time streaming)

It seems slightly more configurable with regards to keep alives then ENet but it doesn't give you as many options (i.e. all data is reliable and sequenced not just the bits that you decide should be). It looks fairly straight forward to implement.

Len Holgate
  • 21,282
  • 4
  • 45
  • 92
9

As others have pointed out, your question is very general, and whether or not something is 'faster' than TCP depends a lot on the type of application.

TCP is generally as fast as it gets for reliable streaming of data from one host to another. However, if your application does a lot of small bursts of traffic and waiting for responses, UDP may be more appropriate to minimize latency.

There is an easy middle ground. Nagle's algorithm is the part of TCP that helps ensure that the sender doesn't overwhelm the receiver of a large stream of data, resulting in congestion and packet loss.

If you need the reliable, in-order delivery of TCP, and also the fast response of UDP, and don't need to worry about congestion from sending large streams of data, you can disable Nagle's algorithm:

int opt = -1;
if (setsockopt(sock_fd, IPPROTO_TCP, TCP_NODELAY, (char *)&opt, sizeof(opt)))
  printf("Error disabling Nagle's algorithm.\n");
smo
  • 913
  • 7
  • 11
  • As I said, assuming TCP is at one end of the scale and UDP at the other, what else is there. – Len Holgate Sep 21 '08 at 18:28
  • 1
    If you want to be pedantic, most of the discussed protocols are built on top of UDP. – smo Sep 22 '08 at 14:06
  • 1
    The assumption that TCP is at one end and UDP is at the other end is false. e.g. UDP has no flow control, you can easily send packets too fast, causing a router inbetween to drop all of them. Then what do you do ? Ignore the lost packets or resend them ? Resending them and you'll end up reimplementing TCP more or less. Another option for reliable communication is SCTP. – nos Jun 23 '09 at 08:11
  • 1
    A fast response doesn't necessarily equal a high throughput. – hookenz Dec 01 '11 at 23:55
  • 1
    I disagree. When nagle is used on TCP based protocols with lots of smaller packets, it'll merge them together and create more larger packets. It causes some slight delay in sending so latecy may increase very slightly. However, throughput can be lower with nagle off because more packets = more packet headers = greater overhead. Packets being dropped on a LAN is usually more to do with input buffers filling. If you have lots of clients sending data to the same host it may make zero difference. I don't believe turning off and on nagle is going to effect it in practice. – hookenz Dec 02 '11 at 00:01
  • One problem with Nagle's algorithm is that while it's fine in isolation, some systems try to avoid acknowledging a packet if they think another packet is going to follow it; since acknowledging the second packet would implicitly acknowledge the first, traffic could be reduced without affecting bandwidth in the case where the second packet does indeed handle the first. Unfortunately, if the sender is using Nagle's algorithm, the recipient is using delayed acks, and after outputting some data the sender wants to receive some data in reply, a semi-deadlock state can arise with... – supercat Jan 29 '14 at 23:31
  • ...both sides of the connection waiting for the other to do something. In practice, something will time out and things will get going again, but the extra lag can often be very noticeable even on machines which have a fast connection between them. – supercat Jan 29 '14 at 23:32
4

If you have a situation where a TCP connection is potentially too slow and a UDP 'connection' is potentially too unreliable what do you use? There are various standard reliable UDP protocols out there, what experiences do you have with them?

The key word in your sentence is 'potentially'. I think you really need to prove to yourself that TCP is, in fact, too slow for your needs if you need reliability in your protocol.

If you want to get reliability out of UDP then you're basically going to be re-implementing some of TCP's features on top of UDP which will probably make things slower than just using TCP in the first place.

17 of 26
  • 27,121
  • 13
  • 66
  • 85
  • Yes, Andrew Edgecombe said as much, but, as I said, I'm interested in the pros and cons of WHAT alternatives there are. Without that list of alternatives and their pros and cons then it's hard to decide what's best. – Len Holgate Sep 21 '08 at 17:16
  • Given a known reliablity function, sometimes a UDP stream can be hand-tuned to outrace the TCP stream in hte OS. Rare though. – Joshua Jan 24 '09 at 21:27
  • 1
    @17 of 26, I agree to Len Holgate, TCP will be slower than reliable UDP in some circumstances. Like high BDP network, suppose you have 1 Gbps internet connection from China to NewYork, I am sure TCP will suck to use almost all of 1 Gbps speed. TCP is better for most of connections on earth, but not for network with High Bandwidth Delay Product. – nullptr Nov 14 '14 at 14:11
4

Protocol DCCP, standardized in RFC 4340, "Datagram Congestion Control Protocol" may be what you are looking for.

It seems implemented in Linux.

bortzmeyer
  • 34,164
  • 12
  • 67
  • 91
3

May be RFC 5405, "Unicast UDP Usage Guidelines for Application Designers" will be useful for you.

bortzmeyer
  • 34,164
  • 12
  • 67
  • 91
2

Did you consider compressing your data ?

As stated above, we lack information about the exact nature of your problem, but compressing the data to transport them could help.

philant
  • 34,748
  • 11
  • 69
  • 112
2

RUDP. Many socket servers for games implement something similar.

Engineer
  • 8,529
  • 7
  • 65
  • 105
1

It is hard to give a universal answer to the question but the best way is probably not to stay on the line "between TCP and UDP" but rather to go sideways :).

A bit more detailed explanation:

If an application needs to get a confirmation response for every piece of data it transmits then TCP is pretty much as fast as it gets (especially if your messages are much smaller than optimal MTU for your connection) and if you need to send periodic data that gets expired the moment you send it out then raw UDP is the best choice for many reasons but not particularly for speed as well.

Reliability is a more complex question, it is somewhat relative in both cases and it always depends on a specific application. For a simple example if you unplug the internet cable from your router then good luck keeping reliably delivering anything with TCP. And what even worse is that if you don't do something about it in your code then your OS will most likely just block your application for a couple of minutes before indicating an error and in many cases this delay is just not acceptable as well.

So the question with conventional network protocols is generally not really about speed or reliability but rather about convenience. It is about getting some features of TCP (automatic congestion control, automatic transmission unit size adjustment, automatic retransmission, basic connection management, ...) while also getting at least some of the important and useful features it misses (message boundaries - the most important one, connection quality monitoring, multiple streams within a connection, etc) and not having to implement it yourself.

From my point of view SCTP now looks like the best universal choice but it is not very popular and the only realistic way to reliably pass it across the Internet of today is still to wrap it inside UDP (probably using sctplib). It is also still a relatively basic and compact solution and for some applications it may still be not sufficient by itself.

As for the more advanced options, in some of the projects we used ZeroMQ and it worked just fine. This is a much more of a complete solution, not just a network protocol (under the hood it supports TCP, UDP, a couple of higher level protocols and some local IPC mechanisms to actually deliver messages). Since a couple of releases its initial developer has switched his attention to his new NanoMSG and currently the newest NNG libraries. It is not as thoroughly developed and tested and it is not very popular but someday it may change. If you don't mind the CPU overhead and some network bandwidth loss then some of the libraries might work for you. There are some other network-oriented message exchange libraries available as well.

mrKirushko
  • 60
  • 1
  • 6
1

You should check MoldUDP, which has been around for decades and it is used by Nasdaq's ITCH market data feed. Our messaging system CoralSequencer uses it to implement a reliable multicast event-stream from a central process.

Disclaimer: I'm one of the developers of CoralSequencer

rdalmeida
  • 1,806
  • 12
  • 16
0

The best way to achieve reliability using UDP is to build the reliability in the application program itself( for example, by adding acknowledgment and retransmission mechanisms)

kushan singh
  • 31