C++ TCP socket sending speed

Question

I'm sending messages to remote server using simple locking TCP socket and the problem I have is that for each message it takes very different time to send it.

And here what I get (some example):

Bytes Sent:  217, Time:  34.3336 usec
Bytes Sent:  217, Time:   9.9107 usec
Bytes Sent:  226, Time:  20.1754 usec
Bytes Sent:  226, Time:  38.2271 usec
Bytes Sent:  217, Time:  33.6257 usec
Bytes Sent:  217, Time:  12.7424 usec
Bytes Sent:  217, Time:  21.5912 usec
Bytes Sent:  217, Time:  31.1480 usec
Bytes Sent:  218, Time:  28.3164 usec
Bytes Sent:  218, Time:  13.0963 usec
Bytes Sent:  218, Time:  82.8254 usec
Bytes Sent:  218, Time:  13.0963 usec
Bytes Sent:  227, Time:  30.7941 usec
Bytes Sent:  218, Time:  27.9624 usec
Bytes Sent:  216, Time:   2.1237 usec
Bytes Sent:  218, Time:  12.3884 usec
Bytes Sent:  227, Time:  31.1480 usec
Bytes Sent:  227, Time:  88.4887 usec
Bytes Sent:  218, Time:  93.0901 usec
Bytes Sent:  218, Time:   7.7870 usec
Bytes Sent:  218, Time:  28.3164 usec
Bytes Sent:  227, Time:  89.5505 usec
Bytes Sent:  218, Time:  84.2412 usec
Bytes Sent:  218, Time:  13.8042 usec
Bytes Sent:  227, Time:  99.4612 usec
Bytes Sent:  218, Time:  86.0110 usec
Bytes Sent:  218, Time:  12.3884 usec
Bytes Sent:  218, Time:  87.7807 usec
Bytes Sent:  216, Time:   3.5395 usec
Bytes Sent:  218, Time:   4.6014 usec
Bytes Sent:  218, Time:  36.1034 usec
Bytes Sent:  218, Time:  14.8661 usec
Bytes Sent:  218, Time:  24.0689 usec
Bytes Sent:  218, Time:  18.0517 usec
Bytes Sent:  227, Time:  24.4229 usec

Does anybody know why this can happen? Why for one message it takes 3 usec to be sent and for other 80 usec?
And is there any way to fix this?

Note: the main goal I want to archive is to send each message as fast as possible. I don't need anychronious sockets, At least until they work faster.

Some additional details regarding what I do:

C++, Visual Studio 2013

How I open:

...
hints.ai_family = AF_UNSPEC;
hints.ai_socktype = SOCK_STREAM;
hints.ai_protocol = IPPROTO_TCP;
...
ConnectSocket = socket(ptr->ai_family, ptr->ai_socktype, ptr->ai_protocol);
...

How I send and calculate time:

...
LARGE_INTEGER cT;
QueryPerformanceCounter(&cT);
long long dT = cT.QuadPart;

iBytesSent = send(ConnectSocket, msgFinal, msgFinalLen, 0);

QueryPerformanceCounter(&cT);
dT = cT.QuadPart - dT;
...

Also I'm listening this socket from other thread, I don't know if this can impact sending or not:

iResult = recv(ConnectSocket, recvbuf, DEFAULT_BUFLEN, 0);

TCP is a stream protocol. It doesn't have concept of "messages", you are sending a byte stream. This means, that receiver might receive the data in different size chunks, than what you send them, for example. TCP only guarantees, that other side receives the same bytes in same order, nothing lost (or there is disconnection if resends fail). — hyde, Aug 18 '15 at 07:47
I don't think you are measuring sending time, you are only measuring `send` call time. — Dariusz, Aug 18 '15 at 07:50
also set TCP_NODELAY if you want your message to be sent out as fast as you can. https://msdn.microsoft.com/en-us/library/windows/desktop/ms740476%28v=vs.85%29.aspx — dau_sama, Aug 18 '15 at 08:15
@dau_sama Except that will actually make the send process *slower*. The `send` operation takes longer when it sends a packet than when it doesn't. Setting TCP_NODELAY will make *every* send operation send a packet. — David Schwartz, Aug 05 '16 at 01:19

score 5 · Answer 1 · edited Apr 01 '20 at 07:52

5

Your methodology is invalid. You're only measuring how long it takes to put the data into the send buffer. If there's room for it, there is no network operation at all. If there isn't room, you block until there is room, which depends on the receiver reading what's already there. So what you're seeing is that sometimes there is room and sometimes there isn't, depending on whether the receiver is reading and keeping up.

If you want to measure round-trip time you need to send a timestamp and have the peer echo it back, then when you receive it compare it to the current time.

edited Apr 01 '20 at 07:52

schoetbi

12,009
10
54
72

answered Aug 18 '15 at 07:50

user207421

305,947
44
307
483

Is there any way to monitore and control what was sent and what is in buffer? Maybe there any recomendations/options/configurations I can specify to make sending process faster and more stable? Or maybe it will help to increate buffer size if its possible? :) – Artem Aug 18 '15 at 08:24
1

@Artem There is a simple way - don't use TCP. TCP is a stream protocol, and it has some guarantees and drawbacks you will find bothersome if you try to use it for sending quick, small packets of data. UDP might be a better choice (though you probably want to use some higher-level library to handle this). But forget about *micro*seconds. In any reasonable scenario, you're still looking at *milli*seconds - it's barely possible to go under a millisecond on a well designed LAN, but outright silly over the internet. – Luaan Aug 18 '15 at 09:02
@Artem **I have already suggested the correct methodology.** What is or isn't in the buffer is irrelevant. – user207421 Aug 18 '15 at 09:42
what if you use echo from terminal instead is that gonna be almost same performance so i dont have to write my own code to test? – Lightsout Jun 25 '18 at 21:46

score 1 · Answer 2 · answered Aug 18 '15 at 07:51

You are not measuring the time it takes for the message to be "sent", you are measuring the time it takes for the message to make it into the TCP send buffer. That could involve memory allocation, lock contention and many other things. It can also include another process being scheduled, leading to you losing your time slice.

score 1 · Answer 3 · edited May 23 '17 at 12:14

What you are measuring is the amount of time send call takes. It is basically a write (I/O) operation into a buffer at socket layer.

Your process attempts i/o and gets blocked - once the i/o finishes the daemon is woken up. The time diff you see for send calls includes:

i. Actual write time.

ii. interruptible sleep time - since the scheduler will not wake up your process immediately after the I/O is done. There may be another process that may be woken up.

Tweaks:

i. Try to optimize the send/receive window size. This is the amount of data that can be sent without waiting for an ACK.

Visit: Setting TCP receive window in C and working with tcpdump in Linux

ii. The buffers you pass on to the send call must fit the window size. So tcp doesn't wait for an OPTIMUM size to be reached before actually flushing the data on the network.

iii. A flag similar to TCP_NODELAY for your OS implementation will help.

iv. Readjust the nice value for your daemon so it is woken up as soon as the blocking I/O call is done.

Thank you, I'd like to clarify one point. "iv. Readjust the nice value for your daemon so it is woken up as soon as the blocking I/O call is done." Could you please explain what does "Readjust the nice value" mean and how this "readjustment" can be done? — Artem, Aug 18 '15 at 15:16
Please visit: http://stackoverflow.com/questions/7618291/difference-between-nice-and-setpriority-in-unix. This will help in getting scheduler's attention. — Abhinav, Aug 19 '15 at 01:53

C++ TCP socket sending speed

3 Answers3