1

I have (I believe) a very classic problem with memory allocation using "new". Here is the piece of code I use:

float * _normals = NULL;
try{

    _normals = new float[_x*_y*_z*3];
}catch(System::Exception ^ e){

    Windows::Forms::MessageBox::Show("Normals:\n" + e->Message);

    if(e->InnerException != nullptr && e->InnerException->Message != nullptr)
        Windows::Forms::MessageBox::Show("Details:\n" + e->InnerException->Message);

    _file->Close();

return 0;
}

I don't know if you can tell from this piece of code but this is a mixed of managed and unmanaged code program. I don't know if that matters.

Now, when I run this code and try to allocate, say, 256*256*128*3 floats it runs normally. When I go with 492*492*442 floats then it throws a "External component has thrown an exception" exception. This is around 1.2GB, right. My system has 6GB of ram and free around 3GB. Can you tell the problem from this information? Can I handle it? I read somewhere about program memory space. Maybe the program memory is not enough?(I don't know anything around that matter, If you can enlighten me)

Please ask if you need more information.

Thank you in advance

C. K. Young
  • 219,335
  • 46
  • 382
  • 435
user3578572
  • 21
  • 3
  • 3
    Are you on 64 bit? 32 bit cannot address more than 3 GB by default. – László Papp Apr 27 '14 at 15:50
  • @LaszloPapp Don't you mean 4GB? – keyser Apr 27 '14 at 15:51
  • KEYSER: [I do not](http://en.wikipedia.org/wiki/3_GB_barrier), but either way, 32 bit could be the reason. – László Papp Apr 27 '14 at 15:51
  • Note that this is C++/CLI, not C++. –  Apr 27 '14 at 15:52
  • @LaszloPapp I see what you mean: ([3 GB barrier](http://en.wikipedia.org/wiki/3_GB_barrier)). Though, it's system dependent, and not even an issue on some systems :p Still, good comment :) – keyser Apr 27 '14 at 15:53
  • What are the values of `_x` ...? What is `catch(System::Exception ^ e) ` – Ed Heal Apr 27 '14 at 15:54
  • @EdHeal It's C++/CLI, as delnan said (I've edited the post accordingly). – C. K. Young Apr 27 '14 at 15:55
  • @LaszloPapp No, `^` is a C++/CLI managed reference. – C. K. Young Apr 27 '14 at 15:55
  • 1
    @ChrisJester-Young: already removed my comment, although without seeing it is c++-cli (whatever that is), I would have assumed typo as the two characters (& and ^) are next two each other on my keyboard. ;) – László Papp Apr 27 '14 at 15:56
  • It's 64bit system. The values for _x,_y,_z when the exception is not thrown is 256,256,128 and when the exception is thrown 496,496,442. Even with the big numbers this is around 1.2GB so even if I had a 32bit system this should work, but it's not. I don't understand the question "What is catch(System::Exception ^e)". I think it's self-explanatory. Oh, and it is C++/CLI, sorry about that. – user3578572 Apr 27 '14 at 15:57
  • @LaszloPapp C++/CLI could perhaps be described as "C++ for .NET", but yes, I can understand why it could be seen as a typo if you've never seen C++/CLI before. – C. K. Young Apr 27 '14 at 15:57
  • @user3578572 It's only self-explanatory if you've seen C++/CLI before and the reader knows you're using it. It's not correct C++. – C. K. Young Apr 27 '14 at 15:58
  • 1
    The more interesting question is why allocating such a huge chunk of memory. Perhaps the algorithm needs to be looked at – Ed Heal Apr 27 '14 at 15:58
  • @user3578572: you wrote 3 GB free memory, so you have 3 GB already used which is the limit for addressing, so 32 bit could have very well be the reason, but as EdHeal is writing, allocating such a big memory is most likely bad idea. – László Papp Apr 27 '14 at 15:59
  • It's 64bit system. I need to allocate this in one go because I need it as an input for an OpenGL function. This is NEED to be in one go allocation. The bytes need to be in continuous memory. – user3578572 Apr 27 '14 at 16:01
  • 2
    That this code runs on a 64-bit operating system is not relevant. What matters is whether you actually built your code with a 64-bit compiler. It will not be with default project settings and the fact that managed code can run in either mode is not relevant for C++/CLI since it produces mixed-mode assemblies. The *new* operator is not managed code. – Hans Passant Apr 27 '14 at 16:07
  • @user3578572: why would it need to go in one shot? Perhaps, you could elaborate more about the use case? – László Papp Apr 27 '14 at 16:09
  • It is indeed configured to build as a 32bit application but, if the memory that I try too allocate is below 2GB do I really need to build it for 64bit? @Laszlo Papp: I need to use the _normals array as input for an OpenGL funcion. This is graphics data that will be uploaded in graphics card. OpenGL needs continuous memory arrays even when we are talking about 2D or 3D arrays. This is, theoritically, a 3D arrays but I need to allocate this in 1D continuous memory array. – user3578572 Apr 27 '14 at 16:18
  • You are contradicting yourself. You mention 3 GB reserved, and then now 2 GB. What is the truth now? – László Papp Apr 27 '14 at 16:19
  • @Laszlo Papp: Really? Please give a little more attention. 3GB free SYSTEM memory. <2GB memory I TRY to allocate. The 3GB reserved memory is memory from OTHER IRRELEVANT programs running on my system. This means it can fit in the free SYSTEM memory and it is less than the 3GB barrier of the 32BIT SYSTEM. – user3578572 Apr 27 '14 at 16:24
  • @Laszlo Papp: No, I want to allocate 1.2GB with 32bit compilation and I will have 4.2GB alltogether on a 64bit system. I think this is valid. Or what? 32bit programs cannot run on 64bit system when the overall system's memory exceeds the 3GB barrier? – user3578572 Apr 27 '14 at 16:28
  • @Laszlo Papp: I re-read it. I still don't understand. Please explain. My system is 64bit. My program is 32bit. I try to allocate memory. This memory is less than 3GB. But when I sum it with the current system's(not program's) allocated memory exceeds the 3GB. Is this a problem? Is this what you are saying? – user3578572 Apr 27 '14 at 16:36
  • Actually, Hans may be wrong here, and got misled by him, sec... 2 GB seems to be the limit for a 32 bit process. Are you exceeding that by any means? – László Papp Apr 27 '14 at 16:38
  • As I said, no. I try to allocate less than 2GB. Specifically 1.2GB – user3578572 Apr 27 '14 at 16:40
  • Well, the exception disagrees with you, so you do not tell the right thing somewhere. I do not think you have provided a reproducable example just yet, but it probably does not matter anyway, as you really should forget allocating 1.2 GB for said reasons. – László Papp Apr 27 '14 at 16:42
  • I see. Maybe the current allocation is less than 3GB but if the overall PROGRAM'S memory exceeds the 3GB I will have a problem? And I really need this allocation to be in one go. – user3578572 Apr 27 '14 at 16:45
  • @user3578572: Memory allocated in other processes doesn't consume any address space in your program. So you are correct that it is irrelevant. The only way it matters is that when you run out of physical RAM, the computer will start using pagefile. That would cause an extreme slowdown, but not an exception. – Ben Voigt Apr 27 '14 at 17:17
  • Running out of memory with allocation would not cause exception? :O I must have missed something the last many years... – László Papp Apr 30 '14 at 05:15

1 Answers1

2

Address space for a 32-bit Windows program (Windows is implied by C++-CLI) running on a 64-bit operating system is either

  • 2 GB by default
  • 4 GB if linked with /LARGEADDRESSAWARE. This flag can also be added later by editbin.

Your problem is address space fragmentation. Just because you've only allocated, say 100MB, doesn't mean that you can allocate another 1.9GB chunk in a 2GB address space. Your new allocation needs to have contiguous addresses.

If, for example, a DLL used by your non-LAA process had a load-address at 0x40000000, then you could allocate a 1GB block below it, or an almost-1GB block above it, but you could never allocate a single block larger than 1GB.

The easiest solution is to compile as 64-bit. Even though the address space will still be fragmented, the open spaces between allocations will be much larger and not cause you problems.

Ben Voigt
  • 277,958
  • 43
  • 419
  • 720
  • `and not cause you problems.` -> that is wrong, it can cause similar issues. The real solution is what I suggested, refactoring the algorithm. – László Papp Apr 27 '14 at 17:02
  • @LaszloPapp: I don't believe he's passing it all to OpenGL in a single chunk. But in order to avoid discontiguities in smaller subsets, the whole thing needs to be continuous. – Ben Voigt Apr 27 '14 at 17:04
  • OK, that is worth of my -1, sorry (and no, I did not give the previous, no matter how you accuse me with childish revenge, etc). Inconsiderately allocating such huge memory areas when they do _not_ have to be continous like that is a very bad practice. – László Papp Apr 27 '14 at 17:04
  • @LaszloPapp: Also, you're suggesting a potential problem which is not the address space. I'm completely correct that in a 64-bit process, address space fragmentation will not cause a problem. – Ben Voigt Apr 27 '14 at 17:05
  • @LaszloPapp: Please explain how to make a 1.2GB discontiguous allocation such that any 1kB subset is contiguous. It can't be done. (Proof by induction shows that the entire block must be contiguous) There are definite advantages to having a single contiguous allocation, even if the data will only be processed in overlapping batches. – Ben Voigt Apr 27 '14 at 17:09
  • @LaszloPapp: BTW, if you do need that much memory allocated in a 64-bit process, your program will run faster if you do it in a single allocation. – Ben Voigt Apr 27 '14 at 17:11
  • @LaszloPapp: how can you say "when they do not have to be continuous"? So do I imagine that? With OpenGL you don't have any other choice. Ben Voigt 52: I think this is it. I will try it after work and post an update. – user3578572 Apr 27 '14 at 18:26
  • Are you really trying to claim that OpenGL was designed by newcomers only allowing to upload data in gigabytes with contiguous area? They would have probably received a lot of complaint if it had been soon... Luckily, it is not. – László Papp Apr 27 '14 at 18:48
  • @LaszloPapp: I commonly have a large data set that I want to scroll through smoothly. When that happens, there will be a scroll offset and viewport width that defines a subrange of the data which will be sent to OpenGL. For the subrange to occupy contiguous storage (which is not precisely necessary, one can split data across multiple VBOs, or load a VBO piecemeal) it is necessary that the entire set be contiguous. Else I will encounter the discontiguity while scrolling through. It seems the OP's case is similar. – Ben Voigt Apr 27 '14 at 18:57
  • That was it. I used the /LARGEADDRESSAWARE option and the memory allocated and uploaded to vram just fine! Thank you. @Laszlo Papp You really need to listen to others and respond accordingly and if you don't know the answer just don't answer. And remember that if someone post to ask a question, that not mean he's stupid or ignorant. – user3578572 Apr 28 '14 at 00:11
  • @AndrewMedico: Close. On a 64-bit OS, applications with `/LARGEADDRESSAWARE` have a 4GB/0GB split (because the kernel uses 64-bit addresses, its private data don't overlap the low 4GB region (actually -2GB to +2GB) at all – Ben Voigt May 17 '14 at 02:45