15

When a sender needs to multicast a relatively large volume of data (say several megabytes per second) in a reliable way over Ethernet to a modest number of receivers (say less than a dozen) on the same subnet, what is the most efficient protocol? By reliable I mean that if a packet is lost, the protocol ensures that it gets resent such that there is no data loss in any receiver. The term efficient is a lot harder to define, but let's say we want to maximize throughput and minimize network bandwidth with modest CPU usage on both ends. That's still not a clear-cut definition but it's the best I can come up with. Either a stream-oriented or a message-oriented protocol would be acceptable.

I'd appreciate real-world examples and I'll gladly accept subjective answers, i.e. what's your favorite multicast protocol, if you can explain its pros and cons.

dbush
  • 205,898
  • 23
  • 218
  • 273
JayMcClellan
  • 1,601
  • 1
  • 11
  • 10
  • How many receivers? If you have less than ten receivers, then you can consider several node-to-node communications. – mcoolive Jul 10 '15 at 08:19

7 Answers7

22

Real-world example: TIBCO Rendezvous.

Data is sent out via multicast with a sequence number. A client that detects a missing sequence number sends out a messge on the multicast group "hey, I missed packet 12345". The server re-multicasts out that data. The server has a configurable amount of data to buffer in case a client requests it.

The problem:

Imagine having a single client that drops half of his packets, and 100 healthy clients. This client sends a retransmission request for every other packet. The server begins to cause enough load on one of the healthy clients such that it starts dropping packets and requesting retransmissions. The extra load from that causes another healthy client to begin requesting retransmissions. And so on. A congestion collapse results.

Tibco provides a workaround, of cutting off a subscriber that sends too many retransmission requests. This makes it harder for a single subscriber to cause a congestion collapse.

The other workaround to limit the risk of congestion collapse is to limit the amount of data that a server is willing to retransmit.

Tibco should also provide heuristics in the client and server as to whether to multicast or unicast the retransmission request, and the retransmission itself. They don't. (For the server, you could unicast the retransmission if only one client requested it in a certain time window, for the client you could unicast the retransmission request if the server has told you - in the retransmitted packet - that you are the only one requesting retransmissions and to please unicast the requests in the future)

Fundamentally you will have to decide between how strongly you want to guarantee that clients receive data vs the risk of congestion collapse. You will have to make guesses as to where a packet was dropped and whether the retransmission is most efficiently sent unicast or multicast. If the server understands the data and can decide to not send a retransmission if there is updated data to be sent anyway (that makes the retransmission irrelevant), you are in a much better position than a framework such as Tibco RV.

Sometimes understanding the data can lead to wrong assumptions. For example, market data - it may seem at first OK to not retransmit a quote when there is an updated quote. But later, you may find that a subscriber was keeping a quote history, not just trying to keep track of the current quote. Perhaps you may have different requirements depending on the subscriber, and some clients will prefer unicast TCP vs multicast.

At some point you will need to make arbitrary decisions on the server of how much data to buffer in case of retransmissions or slow clients.

chuck
  • 922
  • 6
  • 6
  • +1 very clear succinct explanation. As I understand Tibco uses PGM protocol and from OpenPGM website, it's clear that it provides reliable multicasting. However, I'm not sure if it can guarantee low latency as well. Is it possible to have realtime reliable multicast? If not with PGM, is there any available technology out there? Many thanks. – chepukha Mar 01 '12 at 05:19
  • Yes Tibco provides low latency. A typical use case is to broadcast financial values on many screens in a same room (the same LAN). Because of the risk of multicast storm (as explained above) we like to cut the network into segregated multicast zones. It's not a young product, now there are other solutions. – mcoolive Jul 10 '15 at 07:57
5

Following on from TIBCO, the PGM protocol is an open standard reliable multicast with many optimisations to efficiently work at very large scales with network element acceleration. PGM was developed by TIBCO and CISCO and is an optional protocol underneath TIBCO Rendezvous, the default protocol being TRDP which is very similar in design.

You can calculate theoretical efficiencies such as listed here for PGM,

http://code.google.com/p/openpgm/wiki/PgmPerformance

Unfortunately real world network elements, NICs and general computer architectures perform a lot less than the theoretical maximums.

Steve-o
  • 12,678
  • 2
  • 41
  • 60
1

http://www.jgroups.org/

Dave
  • 2,546
  • 6
  • 23
  • 29
1

Might I suggest UFTP. It uses a NAK based mechanism to determine which packets to retransmit and has an option for either a fixed transmission rate or congestion control using TFMCC.

Each file is sent in passes, where the first pass transmits the entire file, while subsequent passes only send retransmissions. Each client keeps track of which packets it received and which ones it missed. At particular checkpoints (and at the end of a pass), if the receiver missed any packets since the last checkpoint it will send a NAK listing the packets that were missed. This has the advantage that low-loss receivers will finish before high-loss receivers. UFTP can also be configured to drop receivers whose percentage of NAKs exceeds a certain threshold.

By limiting NAKs to only receivers that have exhibited loss it reduces the risk of congestion collapse, which is the sender getting overwhelmed by receiver feedback.

Disclosure: author of UFTP.

Community
  • 1
  • 1
dbush
  • 205,898
  • 23
  • 218
  • 273
  • Does it work on IPoIB? I constantly get NAKs, so the server never stops resending packets. I see the file on the client side, but as soon as I stop the server, the file gets deleted! – Vahid Noormofidi Mar 22 '18 at 23:29
  • @Vahid I’ve never used IPoIB so I can’t say for sure, but I would imagine it should work just like any IP link. – dbush Mar 22 '18 at 23:35
1

BitTorrent!

No, seriously. You might want to read up on it.

UDP is useful for multicast but it doesn't provide the guarantees you're looking for - BitTorrent will require you to transmit more than one full copy from the original source, but it's still fairly efficient and provides useful guarantees, especially considering how much checksumming is done on each "chunk" of data passed along.

clee
  • 10,943
  • 6
  • 36
  • 28
  • Thanks, I had never considered BitTorrent. It's not going to be suitable for my application but it's an interesting idea anyway. – JayMcClellan Apr 20 '09 at 13:54
  • BitTorrent presents a lot of interesting ideas in network protocols and protocol design. – user109878 May 23 '09 at 10:04
  • 2
    He specifically asked for an efficient protocol for distributing data on a subnet. While BitTorrent is great for sharing large files with multiple peers, it's not efficient in a LAN setting at all. – liwp Feb 03 '10 at 09:25
  • I'm sorry, where's your better alternative? – clee Feb 03 '10 at 11:05
0

I think you should perhaps take a look at Stream Control Transmission Protocol as an alternative to UDP / multicasting if you really want reliable simultaneous transmission to multiple clients.

Alan
  • 13,510
  • 9
  • 44
  • 50
  • Alan: Do you confirm that with this protocol the same information can be transmit from an endpoint toward several clients "simultaneously" ?I'm currently study the possiblity to use it for dispatch financial market data. – Guillaume Paris Jul 08 '14 at 19:23
  • SCTP, and unicast in general, is not efficient for what the OP is trying to do: Transferring "several megabytes per second" to multiple receivers and "to maximize throughput and minimize network bandwidth". Bandwidth consumption grows linearly with the number of receivers, and a 1GbE link would be saturated at 10MB/s with 12 receivers. Using reliable multicast, e.g. PGM, the bandwidth, under ideal circumstances (no packet loss), is completely independent of the number of receivers, i.e. a 1GbE link would be less than 10% saturated when doing 10MB/s. – Evgeniy Berezovsky Jun 16 '15 at 00:21
  • Also, is SCTP any better for multiple clients than, say, TCP? TCP is heavily optimized even in hardware, i.e. NICs. That often means much reduced CPU consumption (something the OP is also interested in), like Rx Coalescing, Checksum offload, and so forth. So in general any newcomer will have a hard time beating that. That said, I hope SCTP and its concepts, especially efficient multiplexing over IP, will find the adoption they deserve. – Evgeniy Berezovsky Jun 16 '15 at 00:27
0

This is an open research question; there are commercial solutions available but which are prohibitively expensive. Good luck.

Mark P Neyer
  • 1,009
  • 2
  • 8
  • 19