4

I am unable to send IP multicast datagrams of size greater than 64 KB (which I need to, for my experiments). I am transferring packets between computers connected directly by 10 Gigabit ethernet links (without any hops in between, confirmed by traceroute).

All computers have Ubuntu 12.04 installed. I changed the limits of read and write buffers by adding the following lines in /etc/sysctl.conf :

net.core.rmem_max=12582912
net.core.wmem_max=12582912
net.core.rmem_default=12582912
net.core.wmem_default=12582912

and verified the changes using sysctl -a (after running sysctl -p). Do I need to restart so that I can see the changes? (I am sharing the machines, so a restart is not always possible).

The MTU of the interface used to send and receive is 9000 Bytes in all the computers. I have been successful in sending packets of size around 60 KB, and for 100 KB packets, capturing using tcpdump reveals that the packets are not even sent and probably dropped by the kernel (I don't see them in the tcpdump trace).

What more do I need to do to be able to transfer large packets (preferably of sizes of the order of 100 MBs)?

Sagar Jha
  • 1,068
  • 4
  • 14
  • 24

2 Answers2

11

A UDP datagram has to fit inside a single IP datagram. The Total Length field in the IP header is 16 bits, so the maximum length (including the IP and UDP headers) is 65535 bytes. The UDP header also has a 16-bit Length field. The UDP Length field includes the UDP header, not the IP header, but since the entire UDP datagram has to fit in the payload of an IP packet, it's constrained by the IP length.

So it's not possible to send a UDP datagram larger than 64KB. Since the minimum sizes of the IP and UDP headers are 20 and 8 bytes, respectively, the actual maximum amount of payload is at most 65507 bytes.

If you need to send larger messages, you need to break it up into multiple datagrams. Or perhaps you should consider using a different transport protocol, such as TCP (unfortunately, this is not possible if you're doing multicast).

IPv6 supports Jumbograms that are larger than 64K. But you can't do it in IPv4.

Barmar
  • 741,623
  • 53
  • 500
  • 612
  • Actually, the RFC does not says that the `Length` field holds the size of the IP header too. And it happens that `sizeof(UDPHeader) == 8`, so the minimum value is 8, and the maximum payload size is `65535-8=65527`. – 3442 Sep 10 '15 at 22:26
  • I am not sure who downvoted. I upvoted to make sure it is at 0. – Sagar Jha Sep 10 '15 at 22:30
  • @KemyLand The Total Length field in the IP header is what counts, since UDP is carried in an IP packet. "Total Length is the length of the datagram, measured in octets, including internet header and data." That's what RFC 791 says. The Total Length of the IP header includes the IP header too. – nos Sep 10 '15 at 23:01
  • @KemyLand I think you're just looking at the UDP Length field, but the UDP datagram has to fit in an IP packet, so the latter is the ultimate limit. – Barmar Sep 10 '15 at 23:05
  • The peer host reassembles the IP packet into a UDP datagram if fragmentation occurs. https://notes.shichao.io/tcpv1/ch10/#ip-fragmentation This answer is wrong. – Randall Jan 17 '17 at 22:40
  • @Creole I'm talking about the maximum size after reassembly. – Barmar Jan 17 '17 at 22:50
  • You say "A UDP datagram has to fit inside a single IP packet". This is wrong. Several ip packets can make up a single UDP datagram. Most IP networks have packets with max size around 1500 bytes. The reassembled UDP datagram itself must be less than 64kb. – Randall Jan 18 '17 at 23:21
  • I corrected it to say an IP datagram. I was trying to avoid the confusion of using the same term for both layers. – Barmar Jan 18 '17 at 23:43
  • @Creole The hardware layer packets are technically called "frames". – Barmar Jan 18 '17 at 23:44
2

At the lowest level, it's impossible.

According to RFC 768 - User Datagram Protocol, the structure of an UDP packet looks something like...

              0      7 8     15 16    23 24    31
             +--------+--------+--------+--------+
             |     Source      |   Destination   |
             |      Port       |      Port       |
             +--------+--------+--------+--------+
             |                 |                 |
             |     Length      |    Checksum     |
             +--------+--------+--------+--------+
             |
             |          data octets ...
             +---------------- ...

                  User Datagram Header Format

This means that the Length field, holding the packet's size, is 16-bits long. Then, we have that 2^16-1 equals 65535, thus that number being the maximum value that the Length field may contain. And, of course, 65535 is just one byte away from being exactly 64KiB. Maybe there are extensions to UDP that may overcome this issue, but I'm not aware of any. Anyway, it happens that the Length field counts the header's size too, so the minimum value for it is 8, and the maximum payload size is 65535-8=65527.

Anyway, I'm not sure if you're using the right protocol, or at least the right data manipulation model. UDP is surely your (only?) best bet for multicasting purposes, but UDP is meant to transfer small packets of data without providing the connection illution that the Transmission Control Protocol creates to overcome the Internet Protocol's unreliable, connection-less way of transferring information. UDP is just a small improvement that adds ports on top of that.

Hope this helps you!

Community
  • 1
  • 1
3442
  • 8,248
  • 2
  • 19
  • 41
  • He's trying to do multicast, which only works with UDP, not TCP. – Barmar Sep 10 '15 at 22:20
  • Not sure it makes a difference but wanted to point out that he never claimed to be using UDP. `IP Multicast` is what he said. – Zan Lynx Sep 10 '15 at 22:22
  • @Barmar: I didn't meant in any way that he shall use TCP, but anyway, I'll edit to avoid problems. – 3442 Sep 10 '15 at 22:22
  • yes, I am using UDP (I am interested in analyzing what percentage of packets are dropped if the file size is too high/ packets are sent at a very high pace). TCP is reliable. – Sagar Jha Sep 10 '15 at 22:26
  • @SagarJha: So, your question is just theoretical? You're not actually *needing* to send `64KB+` UDP packets? – 3442 Sep 10 '15 at 22:27
  • I am sorry if it was not clear. I am using them for my experiments to analyze packet loss rate and throughput etc. – Sagar Jha Sep 10 '15 at 22:28
  • @SagarJha: Well, all this `64KB` limit stuff is just another probe that the fact that the IP suite took so long to change to `v6` was nothing more than a miracle... – 3442 Sep 10 '15 at 22:31
  • Okay. I saw the following link and thought it could be done : http://docs.oracle.com/cd/E18930_01/html/821-2431/abeis.html#glglz While this link describes the same solution, it is specific for TCP : http://www.cyberciti.biz/faq/linux-tcp-tuning/ – Sagar Jha Sep 10 '15 at 22:35