25

I have like a million records each of about 30 characters coming in over a socket. Can I read all of it into a single string? Is there a limit on the string size I can allocate?

If so, is there someway I can send data over the socket records by record and receive it record by record. I dont know the size of each record until runtime.

aks
  • 1,321
  • 5
  • 15
  • 27

8 Answers8

37

To answer your first question: The maximum size of a C++ string is given by string::max_size

Goutham
  • 819
  • 6
  • 10
  • 8
    Though you might want to mention that in most implementations this maximum size essentially constitutes all memory available to the process. – Omnifarious Sep 06 '10 at 07:22
  • When I saw max_size, I mistook it to be the max_size being set by the user, so a bit more helpful to see it as a program: http://stackoverflow.com/questions/1521281/what-are-the-stl-string-limits?rq=1 – Nav Mar 28 '13 at 11:39
  • It didn't really make much sense because when I used `string::max_size()`, it returned 4611686018427387897, which when I add commas is: 4,611,686,018,427,387,897. This is about 4 _quintillion bytes_. – Victor Resnov Mar 14 '19 at 14:59
  • @VictorResnov That is the maximum number of *characters* – rial Oct 10 '21 at 18:19
  • Well, that answers the question but it is not a good explanation (even considering the comments). If each character is a bit then I could hold up to a 1 million terabytes which has no sense on this hardware. What exactly means that number in the C++ documentation? (at the bottom) According to the following source, in the Microsoft C compiler a string "cannot be longer than 2048 bytes" (source => https://learn.microsoft.com/en-us/cpp/c-language/maximum-string-length?view=msvc-170) – Karmavil Dec 13 '21 at 19:41
  • I could not edit the comment on time: max_size() in my hardware returns half of a million terabytes.. anyway I still don't get it – Karmavil Dec 13 '21 at 19:50
12

std::string::max_size() will tell you the theoretical limit imposed by the architecture your program is running under. Other than that, as long as you have sufficient RAM and/or disk swap space, you can have std::strings of huge size. The answer to your second question is yes, you can send record by record, moreover you might not be able to send big chunks of data over a socket at once - there are limits on the size of a single send operation. That the size of a single string is not known until runtime is not a problem, it doesn't need to be known at compile time for sending them over a socket. How to actually send those strings record by record depends on what socket/networking library you are using; consult the relevant documentation.

A. Sarid
  • 3,916
  • 2
  • 31
  • 56
usta
  • 6,699
  • 3
  • 22
  • 39
7

There is no official limit on the size of a string. The software will ask your system for memory and, as long as it gets it, it will be able to add characters to your string.

The rest of your question is not clear.

Daniel Daranas
  • 22,454
  • 9
  • 63
  • 116
5

The only practical limit on string size in c++ is your available memory. That being said, it will be expensive to reallocate your string to the right size as you keep receiving data (assuming you do not know its total size in advance). Normally you would read chunks of the data into a fixed-size buffer and decode it into its naturally shape (your records) as you get it.

Oliver Bock
  • 4,829
  • 5
  • 38
  • 62
5

The size of a string is only limited by the amount of memory available to the program, it is more of a operating system limitation than a C++ limitation. C++/C strings are null terminated so the string routines will happily process extremely long strings until they find a null.

On Win32 the maximum amount of memory available for data is normally around 2 Gigs.

You can read arbitrarily large amounts of data from a socket, but you must have some way of delimiting the data that you're reading. There must be an end of record marker or length associated with the records that you are reading so use that to parse the records. Do you really want read the data into a string? What happens if your don't have enough free memory to hold the data in RAM? I suspect there is a more efficient way to handle this data, but I don't know enough about the problem.

BeWarned
  • 2,280
  • 1
  • 15
  • 22
1

In theory, no. But don't go allocating 100GB of memory, because the user will probably not have that much RAM. If you are using std::strings then the max size is std::string::npos.

Alexander Rafferty
  • 6,134
  • 4
  • 33
  • 55
-1

If we are talking about char* You are limited with smth about 2^32 on 32-bit systems and with 2^64 on (surprise) 64-bit ones

Update: This is wrong. See comments

Alexander Malakhov
  • 3,383
  • 2
  • 33
  • 58
  • 1
    I didn't downvote, but I can understand how somebody might. Despite the nomenclature, I don't know of any "64-bit" system that actually supports anywhere close to 2^64 of memory. Current CPU designs are typically limited to something like 2^42 bytes of memory (and most motherboards to less than that). – Jerry Coffin Sep 06 '10 at 07:46
  • There's also the issue of memory fragmentation - the address range might not be contiguous - and the question whether `operator new`'s internal bookkeeping can support memory blocks this big. – MSalters Sep 06 '10 at 08:00
  • Thanks, guys. Yeah, even virtual memory on Win7 x64 is limited to 8 TB (=2^43): http://msdn.microsoft.com/en-us/library/aa366778(VS.85).aspx#memory_limits. **Btw**, I hope at least on supercomputers I'm right ? – Alexander Malakhov Sep 06 '10 at 08:39
  • 1
    Nope, especially since those are produced in such small numbers. 2^64 bytes of memory is unaffordable - it's 4 billion times more than the 4GB of a typical PC, and would set you back about 400 billion dollar/euro (!). Therefore nobody designs supercomputer boards with 64 address lines when 48 would do. – MSalters Sep 06 '10 at 11:04
  • As a proof of MSalters' words, today's most [powerful supercomputer](http://www.top500.org/site/systems/3154) have "just" 230 TB of RAM, which is between 2^47 and 2^48 – Alexander Malakhov Feb 18 '11 at 16:26
-2

How about send them with different format?

in your server: send(strlen(szRecordServer)); send(szRecordServer);

in you client: recv(cbRecord); alloc(szRecordClient); recv(szRecordClient);

and repeat this million times.

ironhead
  • 1