linux sockets cpp recv() - cant recv fully data via http

Question

I'm making this socket HTTP client (very basic). When recv()'ing response data from example.com it works fine and writes it all to a buffer but when I try to revc any bigger amounts of data it stops at around 1500 bytes.

Right now all I'm trying to do is get the response written into the buffer (headers and all). Not trying to parse anything. But that isn't working. It works for a few iterations but then stops or hangs. I'm asking for help identifying the issue with this receive_response() function that causes these behaviors.

This is the function that revc's the HTTP response:

void tcp_client::receive_response(char *buffer) {
    int bytes_recv = 0;
    int total_bytes_recv = 0;
    for (;;) {
        bytes_recv = recv(sock, &buffer[total_bytes_recv], CHUNK_SIZE, 0);
        if (bytes_recv <= 0) {
            break;
        } else {
            total_bytes_recv += bytes_recv;
        }
    }
}

The main function:

int main(int argc, char **argv) {
    http_client http;
    char response[100000] = {0};
    http.connect_to_host("go.com", 80);
    http.send_request("GET / HTTP/1.1\r\n\r\n");
    http.receive_response(response);
    std::cout << response << std::endl;
    return 0;
}

Thank you

Fun fact: 1500 bytes is right around a common [MTU](https://en.wikipedia.org/wiki/Maximum_transmission_unit). — user4581301, May 16 '21 at 23:49
[minimum reproducible example](https://stackoverflow.com/help/minimal-reproducible-example) would help reproduce issue. — thirdeye, May 17 '21 at 00:35
You need a good knowledge of RFC 2616 and successors to implement HTTP correctly and this isn't it. Neither, at the TCP level, is expecting `recv()` to fill your buffer. It isn't specified to do that. — user207421, May 17 '21 at 03:33

score 0 · Accepted Answer · answered May 16 '21 at 23:43

You seem to expect the server to close the connection after the response is transmitted. A typical HTTP 1.1 server doesn't do that by default; they keep the connection open for further requests, unless the client explicitly asks otherwise via Connection: close header.

So, you receive all the data, and then the next recv call is sitting there, waiting for more data to arrive.

An HTTP 1.1 client is expected to detect the end of response via Content-Length header, or by decoding a chunked response as indicated by Transfer-Encoding: chunked header.

linux sockets cpp recv() - cant recv fully data via http

1 Answers1