4

From time to time I see network related code in legacy source code and elsewhere modifying the receive buffer size for sockets (using setsockopt with the SO_RCVBUF option). On my Windows 10 system the default buffer size for sockets seems to be 64kB. The legacy code I am working on now (written 10+ years ago) sets the receive buffer size to 256kB for each socket.

Some questions related to this:

  1. Is there any reason at all to modify receive buffer sizes when sockets are monitored and read continuously, e.g. using select?
  2. If not, was there some motivation for this 10+ years ago?
  3. Are there any examples, use cases or applications, where modification of receive buffer sizes (or even send buffer sizes) for sockets are needed?
arghol
  • 745
  • 9
  • 21
  • Does this answer your question? [What are SO\_SNDBUF and SO\_RECVBUF](https://stackoverflow.com/questions/4257410/what-are-so-sndbuf-and-so-recvbuf) ie it's __application specific__ - if the buffer fills before being read the sender will be told to slow down TCP/IP or packets dropped UDP. So it depends on how the application's processing takes. – Richard Critten Dec 06 '19 at 15:42
  • 1
    Maybe partly, but not completely. I know what the buffers are and how they work. My question is more about when to use them in practice. – arghol Dec 06 '19 at 15:46
  • We need more context to answer that. Do packets arrive in bursts? How are they processed and can this be interrupted etc? This is probably too much for a SO type Q&A. – Richard Critten Dec 06 '19 at 15:48
  • In my specific example the network traffic load is low to moderate and the incoming data is handled continuously in its own thread. My questions was intended to be more general though. But maybe it is hard to say someting general about this. – arghol Dec 06 '19 at 15:54

2 Answers2

4

Typically receive-buffer sizes are modified to be larger because the code's author is trying to reduce the likelihood of the condition where the socket's receive-buffer becomes full and therefore the OS has to drop some incoming packets because it has no place to put the data. In a TCP-based application, that condition will cause the stream to temporarily stall until the dropped packets are successfully resent; in a UDP-based application, that condition will cause incoming UDP packets to be silently dropped.

Whether or not doing that is necessary depends on two factors: how quickly data is expected to fill up the socket's receive-buffer, and how quickly the application can drain the socket's receive-buffer via calls to recv(). If the application is reliably able to drain the buffer faster than the data is received, then the default buffer size is fine; OTOH if you see that it is not always able to do so, then a larger receive-buffer-size may help it handle sudden bursts of incoming data more gracefully.

Is there any reason at all to modify receive buffer sizes when sockets are monitored and read continuously, e.g. using select?

There could be, if the incoming data rate is high (e.g. megabytes per second, or even just occasional bursts of data at that rate), or if the thread is doing something between select()/recv() calls that might keep it busy for a significant period of time -- e.g. if the thread ever needs to write to disk, disk-write calls might take several hundred milliseconds in some cases, potentially allowing the socket's receive buffer to fill during that period.

For very high-bandwidth applications, even a very short pause (e.g. due to the thread being kicked off of the CPU for a few quanta, so that another thread can run for a quantum or two) might be enough to allow the buffer to fill up. It depends a lot on the application's use-case, and of course on the speed of the CPU hardware relative to the network.

As for when to start messing with receive-buffer-sizes: don't do it unless you notice that your application is dropping enough incoming packets that it is noticeably limiting your app's network performance. There's no sense allocating more RAM than you need to.

arghol
  • 745
  • 9
  • 21
Jeremy Friesner
  • 70,199
  • 15
  • 131
  • 234
  • in TCP datarates are managed by the TCP window. No data needs to be dropped by endpoints. Data may be dropped by routers along the way though. – doron Dec 06 '19 at 16:11
  • @doron data won't be dropped, but *packets* can be; the application-visible-programmer consequence of that will be that TCP data transfers will be slower than they otherwise might be. – Jeremy Friesner Mar 03 '21 at 00:22
  • Data will only be dropped by the IP layer and only if a packet is unroutable. – doron Mar 03 '21 at 09:09
2

For TCP, the RECVBUF buffer is the maximum number of unread bytes that the kernel can hold. In TCP the window size reflects the maximum number of unacknowledged bytes the sender can safely send. The sender will receive an ACK which will include a new window which depends on the free space in the RECVBUF.

When RECVBUF is full the sender will stop sending data. Since mechanism means the sender will not be able to send more data than the receiving application can receive.

A small RECVBUF will work well on low latency networks but on high bandwidth high latency networks ACKS may take too long to get to the sender and since the sender has run out of window, the sender will not make use of the full bandwidth.

Increasing the RECVBUF size increases the window which means the sender can send more data while waiting for an ACK, this then will allow the sender to make use of the entire bandwidth. It does mean that things are less responsive.

Shrinking the RECVBUF means the sender is more responsive and aware of the receiver not eating the data and can back off a lot quicker.

The same logic applies for the SENDBUF as well.

doron
  • 27,972
  • 12
  • 65
  • 103