0

I'm playing around with parser combinators and have some template heavy code (lots of functions templatized over lambdas). When I try to compile it cl.exe runs up to around 660 MB of memory usage (according to Task Manager), then fails with "C1060: compiler out of heap space". I have plenty more memory available on my system (16 GB + swap file, and only half of that is being used by other processes), so I don't understand why I'm getting this error.

I have read all of the other threads about this issue and the MSDN page and tried their solutions.

  • Switching to the 64 bit tool architecture made the problem go away for awhile, but when my project got bigger the error returned, even though cl.exe is still nowhere near either physical or virtual memory limits.
  • /Zm did not have any effect whether I set it high or low, I'm not using precompiled headers anyways.
  • Changing /MP (multi-process compilation), whether enabling it or disabling it, had no effect.

I'm using Visual Studio Community Edition 2019 v16.11.5 and compiling with /std:c++20, if it matters.

I've pasted my code to Godbolt, though I don't know how to use multi-file projects on Godbolt so this is just to share it. If I comment out various sections of code in ExpandBrace.cpp it compiles successfully in MSVC, so I think this is an actual memory issue and is not tied to any error in the code.

I'm pretty sure my project would compile if cl.exe would use more memory before giving up, but I can't find any way to raise it's heap memory limit.

Derek
  • 53
  • 8
  • 1
    Without looking too deep into this, there's one thing to try out: Split bigger .cpp files into two which you compile separately. It's just a custom to have one .hpp and one .cpp, but it doesn't have to be like that. – Ulrich Eckhardt Oct 28 '21 at 15:19
  • 2
    The right kind of weirdness can result in very rapid consumption of memory followed by the program crashing and releasing it all. It could be one mistaken request for an insane amount of storage. It can be over so fast that no human alive could see it happen. – user4581301 Oct 28 '21 at 15:33
  • @user4581301 Is there a way to investigate that possibility further? My gut feeling is that this is not what's happening, but I would like to properly rule it out. – Derek Oct 28 '21 at 18:01
  • No clue, I'm afraid. I mostly use Visual Studio for the debugger and never had to debug Visual Studio itself. – user4581301 Oct 28 '21 at 18:07
  • @UlrichEckhardt I gave this a try and it worked. I don't like this solution, it's ugly and I feel it shouldn't be necessary, however it does confirm that my code works and I can at least use it as a workaround if no better solution comes up. – Derek Oct 28 '21 at 19:35
  • If the heap is too fragmented then no more allocation can be done even when there are still lots of free memory. Besides address space usage includes memory-mapped files, MMIO devices and other things so you may have no more free address for allocation. But have you tried the 64-bit cl.exe in VS2022? – phuclv Dec 19 '21 at 13:26
  • I've been using the 64-bit toolset, although I have not tried VS2022, I've been using 2019. And it is definitely nowhere close to using all of the available heap space. – Derek Dec 20 '21 at 01:38
  • @Derek no, [VS2022 is **the first 64-bit MSVC**](https://learn.microsoft.com/en-us/visualstudio/ide/whats-new-visual-studio-2022?view=vs-2022#performance-improvements). [All the previous version are 32-bit](https://stackoverflow.com/q/2516436/995714) even though they can compile 64-bit code – phuclv Dec 22 '21 at 09:16
  • Just check `cl.exe` and you'll see that it's a 32-bit application and not 64-bit. And obviously in a 32-bit app you have 2GB of space for all the stuffs including memory-mapped files so heap overflow can easily happen, especially after a huge compilation with lots of allocations, resulting in fragmentation. The memory count you see in task manager is just the private working set, not the whole memory consuming thing like shared memory or memory-mapped addresses – phuclv Dec 22 '21 at 09:20
  • That link is about VS itself, however `cl.exe` is a separate application and it is supposed to be 64-bit if you set Project Properties > Advanced > Preferred Build Tool Architecture to 64-bit. This feature exists in VS2019. And I have checked in Task Manager and `cl.exe` is running in 64-bit mode, as expected (and `devenv.exe`, which is VS, is indeed 32-bits). – Derek Dec 22 '21 at 09:30

0 Answers0