12

Let's say I have two threads, T1 and T2.

Thread T1 makes a blocking write() call on a TCP socket S to send a large buffer of bytes B1. The buffer of bytes B1 is so large that (a) the write call blocks and (b) TCP has to use multiple segments to send the buffer.

Thread T2 also makes a blocking write() call on the same TCP socket S to send some other large buffer of bytes B2.

My questions is this:

Does the implementation of TCP on UNIX guarantee that the all bytes of B1 will be sent before all bytes of B2 (or vice versa)?

Or is it possible that TCP interleaves the contents of B1 and B2 (e.g. TCP sends a segment with B1 data, then a segment with B2 data, an then a segment with B1 data again).

PS - I know it is not a good idea to do this. I'm trying to determine whether or not some code which I did not write is correct.

Robert Harvey
  • 178,213
  • 47
  • 333
  • 501
Bruno Rijsman
  • 3,715
  • 4
  • 31
  • 61
  • What OS are we talking about? – Remus Rusanu Sep 21 '09 at 22:39
  • Linux. (And I would be interested if the answer was different for other flavors of Unix, e.g. OpenBSD or OS X). – Bruno Rijsman Sep 21 '09 at 22:42
  • 2
    I don't know about 'Nix systems, but on Windows, TCP interweaves the two write operations together, requiring locks in the code to prevent simultaneous multi-threaded access to the socket to prevent overlapping. – Remy Lebeau Sep 21 '09 at 23:06
  • 1
    I think you've answered your own question. It's poor practice, regardless of whether the target platform has system call atomicity bugs or not. – user207421 Jul 14 '16 at 23:26

1 Answers1

13

It Tries

TL;DR: for the purpose of writing and debugging code, it's safe to assume atomicity, unless your target is a life support system.

It is always going to be bad if a send(2) (same as write(2)) on a tcp socket is not atomic. There is never a good reason to implement a non-atomic write. All versions of Unix and Windows attempt to keep the write atomic, but apparently very few provide a guarantee.

Linux is known to "usually"1. get this right but it has a bug, even in recent kernels. It does attempt to lock the socket but under certain circumstances a memory allocation can fail and a write will be split up. See this IBM blog entry on sendmsg for details. [Link fixed.]

According to those tests, only AIX and Solaris completely passed a thread-stress-test. It is not known if even those systems have failure cases that simply were not uncovered.

1. TL;DR: Almost always, i.e., always except in the presence of a certain error.
DigitalRoss
  • 143,651
  • 25
  • 248
  • 329
  • 2
    The table at the end of that page you link to shows that FreeBSD and OS X don't provide atomic socket sending either (so can we safely assume you're shocked? ;). I'm not sure it's a bug though - I believe this is the salient sentence from that page: *As a big-picture issue, the POSIX/SUSv3 standard developers indicate that atomicity for socket I/O is "unspecified".* – caf Sep 22 '09 at 04:27
  • Si, I guess multi-thread writes aren't, for most people, a realistic test case. Good points. – DigitalRoss Sep 22 '09 at 07:00
  • That link's dead, looks like there's a copy at http://quark.tistory.com/m/post/235 – Allen Luce Feb 09 '15 at 23:19
  • Thanks, link repointed to the Internet Archive's WayBack Machine. – DigitalRoss Jul 14 '16 at 17:49
  • Does POSIX or SUSv3 tell something about `send` atomicity or maybe explicitly marks it unspecified somewhere? Is there a difference between stream and datagram sockets here? I didn't find an unambiguous answer so far. – gavv Jul 14 '16 at 21:27
  • Related: 1) http://stackoverflow.com/questions/4669710/atomic-write-on-an-unix-socket 2) http://stackoverflow.com/questions/1981372/are-parallel-calls-to-send-recv-on-the-same-socket-valid (that answer probably confuses atomicity and thread-safety; I can't find a proof) – gavv Jul 14 '16 at 21:32
  • 1
    @gavv -- datagrams are different, each write creates a new one, so I bet the issue just doesn't apply. And regarding standards, it doesn't matter, writes are reasonably expected to be atomic whether the spec says so or not. – DigitalRoss Jul 14 '16 at 22:55