1

I am trying to define a large dynamic size 2D array in C. When the array size is not very big (e.g. 20,000 x 20,000), memory allocation works perfectly. However, when the array size is greater than 50K x 50K memory allocation fails. Here is the sample code.

long int i = 0;
int **Cf = calloc(30000, sizeof *Cf);
for( i = 0; i < 30000; i++){
Cf[i] = calloc(30000, sizeof *(Cf[i]));}

The What is the maximum size of buffers memcpy/memset etc. can handle? post has explained why this issue occurs. However, it does not say what to do when you need large dynamic 2D arrays in your program. I am sure there should be an efficient way of doing this since many scientific applications need this feature. Any suggestion?

Ash
  • 117
  • 11
  • Not *all* of the memory available on your computer is available as a single allocatable chunk for your program. – Eugene Sh. Jun 21 '17 at 20:48
  • 2
    It's not RAM that matters here but virtual address space. On a 32 bit system, one cannot address more than 4GB of address space (less in practice due to reserved blocks, fragmentation etc). – kaylum Jun 21 '17 at 20:48
  • Possible duplicate of [How big can a malloc be in C?](https://stackoverflow.com/questions/3463207/how-big-can-a-malloc-be-in-c) – kaylum Jun 21 '17 at 20:50
  • TGhere is no 2D array in your code. `int **` is not a 2D array, cannot represent one or point to one. If you want to use a 2D array, use a pointer to a 1D array. – too honest for this site Jun 21 '17 at 21:19
  • @EugeneSh. You are right. But what is the solution if you need to have large arrays in your program? – Ash Jun 21 '17 at 21:20
  • "However, it does not say what to do when you need large dynamic 2D arrays in your program." - Maybe because it is logical to have enough memory available and a platform supporting that much memory? – too honest for this site Jun 21 '17 at 21:21
  • @kaylum Thank you for the link. But I still need a way to define large 2D arrays in my program. – Ash Jun 21 '17 at 21:21
  • It's a rare situation when you absolutely need an array. Sometimes you can have another data structure. And for some reason I smell an XY-problem here. – Eugene Sh. Jun 21 '17 at 21:22
  • @Olaf If I use a pointer to a 1D array, how many elements do you think I can have in my array? Can I handle 40M elements? Regarding to the enough memory available, the article says that the memory allocation can fail because of the other reasons as well. – Ash Jun 21 '17 at 21:22
  • 1
    FWIW: 50k * 50k `long` are 20GB memory on a POSIX64 system, i.e. 2.5G elements. How do you think it is 40M? And why should that not work? And yes, it can fail. And the computer can explode, you can get a cold. your keyboard can break. I don't see your point. As a sidenote: your pointer-pointer stuff uses quite some more memory than a 2D array. – too honest for this site Jun 21 '17 at 21:23
  • And that is no matter of what "I think", but the system you use! Too bad you don't give any relevant information in your question. I tend to agree with @EugeneSh. this is an XY problem. – too honest for this site Jun 21 '17 at 21:29
  • just curious, what are you doing that requires so much memory? – yano Jun 21 '17 at 21:29
  • You *think* you need a large 2d array. Suggest you look at the problem differently. Eg improve your data structures and algorithm to remove that "need". You'd do better to ask about the underlying problem that makes you think you need such a large block of memory. – kaylum Jun 21 '17 at 21:33
  • @Olaf When I said does it work, I meant is this the right way to tackle large size arrays in C. What more information do you want me to add to the question? – Ash Jun 21 '17 at 21:39
  • @yano I am developing an algorithm for load flow analysis in power system. Depending on the system size, I need to build quite few large arrays in the program. – Ash Jun 21 '17 at 21:42
  • @kaylum I agree with you but for the thing I am trying to do, I need a 2D array. Unless I use a 1D array and a pointer to eliminate the need for 2D array. Should check this method as well. – Ash Jun 21 '17 at 21:44
  • oh ok. Well I don't know the details of how virtual memory and swap space work, but if your OS is telling it doesn't have that much memory, upgrade your box or try a different OS. Otherwise, you'll have to implement some kind of buffered system.. perhaps a double buffer or ring buffer where you read in a partial amount of data, operate on it, write it off somewhere (to file, to the network), then read in the next chunk, etc. Treat it as real-time data processing, for which there is no conceivable end (streaming video for instance). – yano Jun 21 '17 at 21:47
  • Then the answer is simple. It can't be done. If you insist on the need for doing something beyond the limits of the system then that's not possible. And don't repeat the *assertion* that other scientific apps do that - that is not based on fact. Other apps do handle large data sets. But it's not with large arrays. – kaylum Jun 21 '17 at 21:48
  • Are you working on a 32-bit or 64-bit machine? If you're using a 64-bit machine, are you compiling your code for 64-bit operation? Have you looked at the values from `ulimit -a`? How much space are you allowed to allocate. On a 32-bit machine, you're on a hiding to nothing. On a 64-bit machine, you can perhaps create multi-gigabyte arrays, if you have enough physical memory. If you don't have enough physical memory, you're likely to get into all sorts of ghastly techniques involving files used as backing store and (very, very, very) slow access via functions that page data into memory. – Jonathan Leffler Jun 21 '17 at 21:52
  • "Unless I use a 1D array and a pointer to eliminate the need for 2D array." - Did you even read my comments? That does not "eliminate the need for a 2D array", but it **is** a 2D array. What you have **is not a 2D array**. And we are no clairvoyants! If you ask if the C standard disallows such large arrays, the answer is "No!". Wrt to the standard you could allocate a 2TiB*2TiB or larger array. **If your implementation and environment support it**. – too honest for this site Jun 21 '17 at 22:00

0 Answers0