How to receive big data with recv() function using c++?

Question

I'm writing a console application using c++ which using sockets and send an HTTP GET request to a server but the response is an html file bigger than 1000000 infact my buffer: char buffer[1000000]; is too small. I need to receive bigger data from the server than the size of buffer.

I use this code but what is the way to receive a bigger response? I'm a beginner in this programming area so please help me with code and explenations thanks:

char buffer[1000000];


    int nDataLength;
    while ((nDataLength = recv(Socket, buffer, 1000000, 0)) > 0) {
        int i = 0;

        while (buffer[i] >= 32 || buffer[i] == '\n' || buffer[i] == '\r') {
            myString += buffer[i];
            i += 1;
        }
    }

    cout << myString << "\n";

@jpo38 yep but 1000000 is the max size permitted for a char array — Silvia B, Apr 03 '16 at 14:05
What `recv` returns? A number or `SOCKET_ERROR` (-1). Small buffer should not be a problem, you should read partial answer for each call to `recv` and end up having all data being received at some point. Please put `cout << "recv returned " << nDataLength << endl`within your loop to see if you received some data or not. — jpo38, Apr 03 '16 at 14:25
@jpo38 `nDataLength` is 0 out of the loop.. within the loop for sometime is equals to 35040 and the last couts are equals to 45617 — Silvia B, Apr 03 '16 at 14:27
So you are getting some data, it's not a problem with your buffer size. What's the final size of `myString`? What makes you tell you did not get the whole request? What's returned by `recv` when you exit the while loop? — jpo38, Apr 03 '16 at 14:39
@jpo38 I don't can see the size of `myString` because the `cout` doesn't show me the size. `myString` contain not all html page of this website: http://www.cookaround.com/cerca?q=pasta I don't know why but `myString` store the html page only until the `.html` css definition — Silvia B, Apr 03 '16 at 14:59
Do `cout << myString.size()` to see the string size. `cout << myString;` may not show the full content if the string contains special characters that could not be redirected to `cout`. — jpo38, Apr 03 '16 at 16:43
I just realized your nested loop does not even check `nDataLength`, so you must be copying data from `buffer` to `myString` that was actually not read by `recv` (when `i` gets greater or equal to `nDataLength`). Check Remy Lebeau answer, that must solve your problem. — jpo38, Apr 03 '16 at 16:49
@jpo38 Thanks so much.. really the code of Remy Lebeau works correctly thanks to your help.. only I hope that this question will vote up... however thanks a lot — Silvia B, Apr 04 '16 at 12:50

Remy Lebeau · Accepted Answer · 2020-05-03T10:04:19.377

3

You need to use a smaller fixed length buffer when reading from the socket, and then append the received data to a dynamically growing buffer (like a std::string, or a file) on each loop iteration. recv() tells you how many bytes were actually received, do not access more than that many bytes when accessing the buffer.

char buffer[1024];
std::string myString;

int nDataLength;
while ((nDataLength = recv(Socket, buffer, sizeof(buffer), 0)) > 0) {
    myString.append(buffer, nDataLength);
}

std::cout << myString << "\n";

edited May 03 '20 at 10:04

answered Apr 03 '16 at 16:17

Remy Lebeau

555,201
31
458
770

append() doesn't store non text data. – ozn May 03 '20 at 09:02
@ozn yes, it does. `std::string` is just a container of *arbitrary* `char`s, it doesn't really care what you store in it. It happily accepts non-textual `char`s, including `nul`. And since 1 `char` is 1 byte, and `recv()` takes a `char[]` buffer, and `append()` takes a `char*` pointer, it all works out fine. – Remy Lebeau May 03 '20 at 10:06
Yeah, you are right. I think I was doing just `myString.append(buffer)`; I suppose this wouldn't work if the buffer is not text based (like an image). – ozn May 07 '20 at 04:40
@ozn the 1-param `append()` requires a null-terminated C string, which indeed does not work for non-textual data. The 2-param `append()` accepts a pointer and length, which works just fine for non-textual data. – Remy Lebeau May 07 '20 at 05:30
Oh, I see. Just wondering, why does it require the null-terminated C string? Would it work if we were to add a null-termination at the end of the char array? – ozn May 07 '20 at 05:57
@ozn how else do you expect it to know the length of the data to append, if you only pass in a pointer to the data and not the length of the data? It has to scan the data looking for a delimiter, which for a `char` string is the null terminator. – Remy Lebeau May 07 '20 at 15:52
Makes sense, I forget that char array doesn't store the length in c/c++. In higher level languages the list like data structure does. My mistake. I also assumed that if I were to just pass `myString.append(buffer)` when the buffer is text, it doesn't look for the length but till the end of the array. Perhaps I forgot to put the null character at end of buffer in that case. – ozn May 08 '20 at 00:40
`myString.append(buffer)` would work fine if `buffer` were a `std::string`, then `append()` would take `buffer.size()` into account. But with a `char[]` array, no. A null terminator is required, unless you pass the array's data length explicitly (which you should, when using `recv()`, since the length may be smaller than the array's total size) – Remy Lebeau May 08 '20 at 00:44
Makes sense. Thanks for the clarification! – ozn May 09 '20 at 01:48

score 1 · Answer 2 · answered Apr 03 '16 at 14:14

recv return value is total size of receved data. so you can know total data size, if your buffer is smaller than total data size there is 2 solutions. I guess... 1. allocate buffer on the heap. using like new, allcoc etc. 2. store received data to data structure(like circular queue, queue) while tatal data size is zero(recv function return)

I prefer to use 2nd solution. Googling about recv function , socket programming sample codes. That'll helpfull.

How to receive big data with recv() function using c++?

2 Answers2