if send() returns x bytes, does recv() get the same amount of bytes in one call?

Question

I'm aware of the need to call send() in a loop until the desired amount of bytes have been sent. Also on the receiving side.

Here is a part of wrapper around recv I wrote:

do{
            result = socket->receiveData(m_recvBuf + recv_len , BUF_LENGTH - recv_len);
            recv_len += result;
.....

I'm a little confused about couple of things, so here goes:

1. if the send() returns 10 bytes, are those ten bytes still only at the sender side, but ready to be sent. Or have the bytes physically arrived to the receiver computer?

2. If the answer to the above is yes, does then calling recv() always return those 10 bytes as they have already arrived?

3. I could also put it this way; if send has been called three times each time returning 10, so total bytes sent supposedly being 30. Does then calling recv(), one time, return 30 bytes?

Question 1. edited as "still only at the receiver side" should be "still only at the sender side".

Scenario: My program in pc1 calls send(); send() returns 1; My code things that one byte has been sent to the receiver program in pc2. The network cable got eaten by a dog the moment after my send() function returned 1.

If that is the case in real life, I surely have misunderstood the benefits of TCP vs UDP.

Thanks all for giving time in answering.

Answer 1

I'll try:

1. You don't know. All that is known is that the network stack has accepted the data, and will do its best to transfer it.
2. N/A
3. No, there are no such guarantees. TCP/IP is stream-oriented, it's two streams of bytes with no further structure (messages, packets, writes, frames, whatever). You must make sure your data serialization format supports finding the message boundaries so the receiver can interpret the data.

Answer 2

if send() returns x bytes, does recv() get the same amount of bytes in one call?

In general, certainly no !!

For example, for TCP/IP sockets (see tcp(7) & socket(7)) going through wifi routers and/or intercontinental routers, packets could be fragmented and/or reassembled. So a given send can correspond to several recv and vice versa, and the "boundaries" of messages are not respected. Hence, for applications, TCP is a stream of bytes without any message boundaries. Read also about sliding window protocol and TCP congestiion control used inside TCP.

In practice, you might observe, e.g. between two computers on the same Ethernet cable, that packets are not fragmented or reassembled. But you should not code with that hypothesis.

Concretely, application level protocols like HTTP or SMTP or JSONRPC or X11 protocols should be designed to define message boundaries and both server and client sides should do buffering.

You'll want to use poll(2), see this answer.

if the send() returns 10 bytes, are those ten bytes still only at the receiver side, but ready to be sent.

It is not easy to define what "being still at the reciever side" really means (because you don't really care about what happens inside the kernel, or inside the network controller, or on intercontinental cables). Therefore the above sentence is meaningless.

Your application code should only care about system calls (listed in syscalls(2)...) like poll(2), send(2) and related, write(2), recv(2) and related, read(2), socket(2), accept(2), connect(2), bind(2) etc...

You might want to use messaging libraries like 0mq.

The network cable got eaten by a dog the moment after my send() function returned 1.

Why do you care that much about such a scenario. Your dog could also have dropen your laptop, or have pee-ed on it. Once send has told your application than ten bytes have been emitted, you should trust your kernel. But the receiving program might not yet have gotten these bytes (on another continent, you'll need to wait dozens of milliseconds, which is a quite big delay for a computer). Very probably, the ten bytes are in the middle of the ocean when your dog have bitten your Ethernet cable (and you can reasonably code as if they have been emitted). The TCP protocol will detect that the link has been interrupted, but that error would be given to your program much later (perhaps as an error for the next call to send happening ten seconds after).

^{(there are some large macroscopic delays in the TCP definition, perhaps as large as 128 seconds -I forgot the details- and these delays are too small for interplanetary communication; so TCP can't be used to Mars)}

You should (most of the time) simply reason at the system call level.

^{(of course, in some cases -think of remote neurosurgical robots- that might not be enough)}

I surely have misunderstood the benefits of TCP vs UDP.

If you just used UDP, a given packet could be fragmented, lost or received several times. With TCP, that cannot reasonably happen (at least when the next packets have been successfully sent and received).

Answer 3

my motto: "if in doubt, try it out".

This is a complete program that demonstrates that on my machine, an entire packet of one million bytes does not even make it through the loopback adapter without being buffered into separate reads (and possibly writes, as I have used the composite function asio::write() :

#include <thread>
#include <mutex>
#include <condition_variable>
#include <boost/asio.hpp>
#include <boost/system/error_code.hpp>
#include <vector>
#include <iostream>

namespace asio
{
    using namespace boost::asio;
    using boost::system::error_code;
}

using protocol = asio::ip::tcp;

struct event
{
    std::mutex mutex;
    std::condition_variable cv;
    bool notified = false;

    void notify()
    {
        auto lock = std::unique_lock<std::mutex>(mutex);
        notified = true;
        lock.unlock();
        cv.notify_all();
    }

    void wait()
    {
        auto lock = std::unique_lock<std::mutex>(mutex);
        cv.wait(lock, [this] { return this->notified; });
    }
};

struct emitter
{
    emitter(std::ostream& os) : os(os) {}

    template<class...Ts>
    void operator()(Ts&&...ts)
    {
        auto lock = std::unique_lock<std::mutex>(m);
        auto emit_item = [&os = this->os](auto&& x)
        {
            os << x;
        };
        using expand = int[];
        void(expand { 0,
            (emit_item(ts),0)...
        });
        os << std::endl;
    }

    std::ostream& os;
    std::mutex m;
};

event rx_ready;
emitter stdout_emit { std::cout };

void sender()
{
    asio::io_service executor;

    auto big_buffer = std::vector<char>(1000000, 'a');
    protocol::socket sock(executor);
    rx_ready.wait();
    asio::error_code ec;
    if(sock.connect(protocol::endpoint(asio::ip::address_v4(0x7f000001), 12345)), ec) {
        stdout_emit("connect failure: ", ec.message());
        return;
    }
    auto written = asio::write(sock, asio::buffer(big_buffer), ec);
    stdout_emit("wrote: ", written);
    if (ec) {
        stdout_emit("write failure: ", ec.message());
    }
    sock.shutdown(protocol::socket::shutdown_send, ec);
    if (ec) {
        stdout_emit("shutdown failure: ", ec.message());
    }
    sock.close(ec);
    if (ec) {
        stdout_emit("close failure: ", ec.message());
    }
}

void start_receiving(protocol::socket& s)
{
    auto huge_buffer_ptr = std::make_shared<std::vector<char>>(1000000);

    s.async_read_some(asio::buffer(*huge_buffer_ptr), [huge_buffer_ptr, &s](asio::error_code ec, std::size_t size)
    {
        stdout_emit("read ", size, " bytes");
        if (ec)
        {
            stdout_emit("read error: ", ec.message());
        }
        else
        {
            start_receiving(s);
        }
    });
}

void receiver()
{
    asio::io_service executor;
    protocol::acceptor acceptor(executor);
    auto ep = protocol::endpoint(protocol::v4(), 12345);
    acceptor.open(ep.protocol());
    acceptor.bind(ep);
    acceptor.listen();

    protocol::socket s(executor);
    acceptor.async_accept(s, [&](asio::error_code ec){
        if (ec) {
            stdout_emit("accept: ", ec.message());
        }
        else
        {
            start_receiving(s);
        }
    });
    rx_ready.notify();
    executor.run();
}

int main()
{
    auto t = std::thread(receiver);

    sender();

    t.join();
}

sample results:

read 393216 bytes
wrote: 1000000
read 606784 bytes
read 0 bytes
read error: End of file

Process finished with exit code 0

changing the read and write buffers to 10,000,000 bytes gave me this:

read 393216 bytes
read 638820 bytes
read 639028 bytes
read 639028 bytes
read 639028 bytes
read 638820 bytes
read 639028 bytes
read 639028 bytes
read 639028 bytes
read 638820 bytes
read 639028 bytes
read 639028 bytes
read 639028 bytes
read 638820 bytes
wrote: 10000000
read 639028 bytes
read 639028 bytes
read 22196 bytes
read 0 bytes
read error: End of file

Process finished with exit code 0

Answer 4

if the send() returns 10 bytes, are those ten bytes still only at the sender side, but ready to be sent. Or have the bytes physically arrived to the receiver computer?

You cannot tell where the 10 bytes are exactly, some of them can be waiting somewhere in the sending machine, some over some wire and some waiting somewhere in the receiving machine.

If the answer to the above is yes, does then calling recv() always return those 10 bytes as they have already arrived?

N/A

I could also put it this way; if send has been called three times each time returning 10, so total bytes sent supposedly being 30. Does then calling recv(), one time, return 30 bytes?

You cannot tell! The only thing you can say, in TCP mode, is that bytes are received in the same order that they have being sent.

Scenario: My program in pc1 calls send(); send() returns 1; My code things that one byte has been sent to the receiver program in pc2. The network cable got eaten by a dog the moment after my send() function returned 1.

Then you cannot say anything...

If that is the case in real life, I surely have misunderstood the benefits of TCP vs UDP.

UDP is datagram-oriented semantic, like postal sytem (no preserved order, no guaranty of any kind, loss is possible, duplicate, etc)

TCP is stream-oriented semantic, like phone system (preserved order and no loss).

Of course in case of hard network failure, TCP cannot ensure anything!

As TCP is build on top of IP (datagram) what is sent over TCP is fragmented to be sent via IP, and you don't control such fragmentation (I forgot to tell about caching, etc).