UDP buffer overflow w/o filling the receive buffer?

Question

If I send 1000 "Hello World!" UDP messages (12 bytes + 28 IP/UDP overhead), I observe that on the receiving side I only buffer 658 (always the same number, 658*40 = 26320 bytes). I do that, by sending the UDP messages while sleeping on the server (after creating the socket).

Curiously the SO_RCVBUF option on the server is 42080 bytes. So, I wonder why I can not buffer the 1000 messages. Do you know where are spend the remaining 15760 bytes?

Below the server code (where distrib.h contains basic error handling wrappers of the socket and signal handling functions):

#include "distrib.h"

static int count;
static void sigint_handler(int s) {
  printf("\n%d UDP messages received\n",count);
  exit(0);
}

int main(int argc, char **argv) 
{
  struct addrinfo* serverinfo;
  struct addrinfo hints;
  struct sockaddr_storage sender;
  socklen_t len;
  int listenfd,n;
  char buf[MAXLINE+1];

  if (argc != 2) {
    log_error("usage: %s <port>\n", argv[0]);
    exit(1);
  }

  Signal(SIGINT,sigint_handler);

  bzero(&hints,sizeof(hints));
  hints.ai_family = AF_INET;    
  hints.ai_socktype = SOCK_DGRAM; 
  hints.ai_protocol = IPPROTO_UDP;
  Getaddrinfo("127.0.0.1", argv[1], &hints, &serverinfo);

  listenfd = Socket(serverinfo->ai_family, serverinfo->ai_socktype, 
            serverinfo->ai_protocol);
  Bind(listenfd, serverinfo->ai_addr,serverinfo->ai_addrlen);
  freeaddrinfo(serverinfo);

  count =0;
  sleep(20); 
  while(true) {
    bzero(buf,sizeof(buf));
    len = sizeof(sender);
    n = Recvfrom(listenfd, buf, MAXLINE, 0, (struct sockaddr*)&sender,&len);
    buf[n]='\0';
    count++;

  }

  close(listenfd);

  return 0;
}

Are you sure that all messages have been sent? Are both server and client programs on the same computer/network? — Some programmer dude, May 08 '13 at 11:47
If you are on Linux, did you check _netstat -su_ [ref](http://stackoverflow.com/questions/10899937/udp-packet-drops-by-linux-kernel) — thuovila, May 08 '13 at 11:47
In addition to the socket recv buffer size there might be a bunch of other settings affecting the number of queued packets.As an example, _net.core.netdev_max_backlog_ is the number of unprocessed input packets that can be present. — thuovila, May 08 '13 at 12:08
@Joachim: yes, all the messages have been sent. I can see it using tcpdump, and the sendto message does not fail. In fact, if I remove the sleep all the UDP messages are received. Regarding the second question, both client and server are in the same PC, the interface being used is therefore the loopback. — marcmagransdeabril, May 08 '13 at 12:11
@thuovilla: If I remove the sleep everything works as expected. — marcmagransdeabril, May 08 '13 at 12:12
@JoachimPileborg: netdev_max_backlog is the max number of unprocessed input packets by the **kernel**. As the server is sleeping the kernel should have as much time as he needs to process them... In any case, I'm using MacOS 10.7.5 and this parameter does not exist there (sysctl -a). — marcmagransdeabril, May 08 '13 at 12:55
BTW, using netstat I observe exactly 342 UDP packets "dropped due to full socket buffers" on the loopback interface. What I do not understand is why I get exactly this number (i.e. exactly 658 messages are kept on the buffers). — marcmagransdeabril, May 08 '13 at 14:02

Chris Dodd · Accepted Answer · 2013-05-08T18:49:25.223

5

It's more informative to do the reverse calculation -- your buffer is 42080 and it's buffering 658 packets before it starts dropping. Now 42080/658 = 63.95, so it looks like it is counting each packet as 64 bytes and dropping packets if the total size of the packets buffered so far is at or above the limit. Since it buffers entire packets, it actually ends up buffering slightly more than the limit.

Why 64 bytes instead of 40? Perhaps it's including some queuing overhead or perhaps it's rounding up to a multiple of some power of 2 for alignment, or perhaps some combination of both.

edited May 08 '13 at 18:49

answered May 08 '13 at 15:19

Chris Dodd

119,907
13
134
226

socket (7) manual also mentions that 50% of the rcvbuf would be reserved for book-keeping, so in theory you shouldn't rely on more than 50% of skb. – wick Mar 10 '21 at 20:42

thuovila · Answer 2 · 2013-05-08T20:44:42.220

I dont have a complete answer, but I tested this on my Linux box and this is what I observed.

When I send one "Hello World!\n" with a terminating '0'. I get:

Client:

$./sendto
sent 14 bytes

Socket "Recv-Q" has 768 bytes (seems probable its in bytes, did not check ss sources):

$ ss -ul|grep 55555
UNCONN     768    0               127.0.0.1:55555                    *:*

When I send 1000 packets I get:

$ ./sendto 
sent 14000 bytes

Recv-Q:

$ ss -ul|grep 55555
UNCONN     213504 0               127.0.0.1:55555                    *:*

Your server (after ctrl-c):

$ ./recvfrom 55555
^C
278 UDP messages received

Incidentally 213504/768 = 278. With quick experimentation I could not figure out what setting to tune, to increase the buffered amount. Also, I dont know why a received packet takes so much space in this queue. Lots of metadata maybe? As on you osX, the dropped packets show up in netstat -su.

EDIT: Additional observation with ss -ulm, which prints "socket memory usage" in more detail:

UNCONN     213504 0               127.0.0.1:55555                    *:*       
skmem:(r213504,rb212992,t0,tb212992,f3584,w0,o0,bl0)

The 213504 bytes buffered are 512 bytes above the rb value. Might not be a coincidence, but would require reading the kernel source to find out.

Did you check how much one UDP datagram takes up on osX?

EDIT 2: This is still not a suitable answer for osX, but on Linux I found that increasing the kernel memory allocated for receive buffers allowed me to buffer all the 1000 packets sent.

A bit of overkill, but I used these (disclaimer) tweaking the buffer values randomly might seriously mess up your networking and kernel):

net.core.rmem_max=1048568
net.core.rmem_default=1048568

UDP buffer overflow w/o filling the receive buffer?

2 Answers2