5

If i have a TCP connection that transfers data at 200 KB/sec but i only read()/recv() from the socket once a second, where are those 200 KB of data stored in the meanwhile?

As much as I know, data leaves the TCP socket's read buffer after an ack gets sent to the sender, and it's too small anyways to hold 200KB of data, where does it wait in the meanwhile until it can be read()/recv() by my client?

Thanks!!

The following answer claims data leaves the TCP read buffer as soon as it is ACK'ed, before being read()/recv()d:

https://stackoverflow.com/a/12934115/2378033

"The size of the receiver's socket receive buffer determines how much data can be in flight without acknowledgement"

Could it be that my assumption is wrong and the data gets ACK'd only after it is read()/recv()d by the userspace program?

Community
  • 1
  • 1
rick
  • 119
  • 1
  • 3
  • 1
    The operating system buffers a certain amount. Once those buffers get full it will tell the sender to stop sending. The sender may then choose to buffer a certain amount. The sending appliction will then be unable to write any more so it will either block or buffer data itsself. – jcoder May 13 '13 at 14:08
  • How much does the OS buffer? how can i access this buffer or set its options like size? i assume you're not refering to the TCP read buffer coz that's only ~ 8 KB in size – rick May 13 '13 at 14:13
  • On the linux machine I'm using they are set using /proc/sys/net/ipv4/tcp_mem and similar - $ cat /proc/sys/net/ipv4/tcp_mem 2097152 2097152 16777216 – jcoder May 13 '13 at 14:55
  • 1
    The answer you cited does not support your claim in any way. – user207421 May 15 '13 at 00:58

3 Answers3

8

data leaves the TCP socket's read buffer after an ack gets sent to the sender

No. It leaves the receive buffer when you read it, via recv(), recvfrom(), read(), etc.

The following answer claims data leaves the TCP read buffer as soon as it is ACK'ed

Fiddlesticks. I wrote it, and it positively and absolutely doesn't 'claim' any such thing.

You are thinking of the send buffer. Data is removed from the sender's send buffer when it is ACKed by the receiver. That's because the sender now knows it has arrived and doesn't need it for any more resends.

Could it be that my assumption is wrong and the data gets ACK'd only after it is read()/recv()d by the userspace program?

Yes, your assumption is wrong, and so is this alternative speculation. The data gets ACK'd on arrival, and removed by read()/recv().

user207421
  • 305,947
  • 44
  • 307
  • 483
2

When data is correctly received it enters the TCP read buffer and is subject to acknowledgement immediately. That doesn't mean that the acknowledgement is sent immediately, as it will be more efficient to combine the acknowledgement with a window size update, or with data being sent over the connection in the other direction, or acknowledgement of more data.

For example suppose you are sending one byte at a time, corresponding to a user's typing, and the other side has a receive buffer of 50000 bytes. It tells you that the window size is 50000 bytes, meaning that you can send that many bytes of data without receiving anything further. Every byte of data you send closes the window by one byte. Now the receiver could send a packet acknowledging the single byte as soon as it was correctly received and entered the TCP receive buffer, with a window size of 49999 bytes because that is how much space is left in the receive buffer. The acknowledgement would allow you to remove the byte from your send buffer, since you now know that the byte was received correctly and will not need to be resent. Then when the application read it from the TCP receive buffer using read() or recv() that would make space in the buffer for one additional byte of data to be received, so it could then send another packet updating the TCP window size by one byte to allow you to once again send 50000 bytes, rather than 49999. Then the application might echo the character or send some other response to the data, causing a third packet to be sent. Fortunately, a well-designed TCP implementation will not do that as that would create a lot of overhead. It will ideally send a single packet containing any data going in the other direction as well as any acknowledgement and window size update as part of the same packet. It might appear that the acknowledgement is sent when the application reads the data and it leaves the receive buffer, but that may simply be the event that triggered the sending of the packet. However it will not always delay an acknowledgement and will not delay it indefinitely; after a short timeout with no other activity it will send any delayed acknowledgement.

As for the size of the receive buffer, which contains the received data not yet read by the application, that can be controlled using setsockopt() with the SO_RCVBUF option. The default may vary by OS, memory size, and other parameters. For example a fast connection with high latency (e.g. satellite) may warrant larger buffers, although that will increase memory use. There is also a send buffer (SO_SNDBUF) which includes data that has either not yet been transmitted, or has been transmitted but not yet acknowledged.

mark4o
  • 58,919
  • 18
  • 87
  • 102
1

Your OS will buffer a certain amount of incoming TCP data. For example on Solaris this defaults to 56K but can be reasonably configured for up to several MB if heavy bursts are expected. Linux appears to default to much smaller values, but you can see instructions on this web page for increasing those defaults: http://www.cyberciti.biz/faq/linux-tcp-tuning/

Mark B
  • 95,107
  • 10
  • 109
  • 188