3

I'm using C++ in Visual Studio 2013 on a Windows 7 x64 OS with 8.00 GB of RAM.

I'm programming a DLL and have a 2001 x 4001 array of const floats for a large lookup table of ~8 million values, which should be 32 MB.

Everything works fine when I compile without initializing the table with values and when I try to compile with initializing approximately 1400 x 4001 of the values.

However, when I try to initialize the entire 2001 x 4001 table, I get:

c1060 compiler is out of heap space error.

I've added /Zm200 (and /Zm1000) to the command line options, but then error becomes:

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets(341,5): error MSB6006: "CL.exe" exited with code 3.

I've tried in Debug and Release and get the same results.

Unfortunately I can't find anything helpful on this specific error.

I'm not sure where to look, but 32 MB doesn't seem to be an inordinate amount of memory and even if so, wouldn't the /Zm1000 have made plenty of memory available if the default /Zm100 was enough for nearly 3/4 of the table?

Finally, the vast majority of the DLL is the table. There are about 150 other lines of code in total but only a handful of standard variables (int, double, etc.) are in scope at any one time.

Here's the definition of the array:

static SKEWLOGISTICFUNCTIONSDLL_API const int shapeValues = 2001; // Enough for 0.000 to 2.000 in increments of 0.001
static SKEWLOGISTICFUNCTIONSDLL_API const int distanceValues = 4001; // Enough for -2.000 to 2.000 in increments of 0.001

static SKEWLOGISTICFUNCTIONSDLL_API const float skewLogisticTable[shapeValues][distanceValues];

And here's the start of the large table:

    const float MySkewLogisticFunctions::skewLogisticTable[shapeValues][distanceValues] =
{
    // Shape going down --> 0.000 to 2.000 in increments of 0.001
    // Distance going across --> -2.000 to 2.000 in increments of 0.001
    { 0.119203, 0.119308, 0.119413, . . .
TheLionKing
  • 49
  • 2
  • 2
    Any feedback on the down vote? – TheLionKing Jan 13 '16 at 18:37
  • Why don't you allocate the memory dynamically ? – Marged Jan 13 '16 at 18:37
  • I can but then it takes an inordinate amount of time to run. I'm trying to get it all in the compile time. – TheLionKing Jan 13 '16 at 18:38
  • How exactly your array definition looks like? `const float arr[bla][blah] {values};`? If so, have you tried `constexpr`? – HolyBlackCat Jan 13 '16 at 18:47
  • Provide your source or at least the relevant lines where you define and initialize – Marged Jan 13 '16 at 18:57
  • 1
    I can guess that 8 million float literals in a source file is not (yet) part of the compiler test suite. Why not store it in a binary file and load that during startup? – Bo Persson Jan 13 '16 at 19:04
  • Added code to the original post. I tried `constexpr` but I get a host of syntax errors. It doesn't appear to support that or I don't have a correct setting to use it. Here's what I had: `static SKEWLOGISTICFUNCTIONSDLL_API constexpr int shapeValues = 2001;` – TheLionKing Jan 13 '16 at 19:04
  • "Why not store it in a binary file and load that during startup?" -- I'm not familiar with that, but I'll look that up. – TheLionKing Jan 13 '16 at 19:05
  • @BoPersson So, if I went the binary file route, I would just tell the program running the DLL to load the binary file on startup? – TheLionKing Jan 13 '16 at 19:17
  • Another crazy idea :-) If the values are *that* regular, why do you have to store them all instead of calculating them as needed? – Bo Persson Jan 13 '16 at 19:24
  • *"I'm trying to get it all in the compile time."* - You'll still pay the memory allocation cost at run time, it'll just be paid when the OS loads the module's pages instead of somewhere under `main`. – MooseBoys Jan 13 '16 at 19:25
  • @BoPersson As the Shape and Distance parameters approach zero, the function that calculates them takes longer, such as a few seconds once one of the parameters is below 0.1 or even for hours if one of the parameters is at zero. My solution is provide a table for the area of concern (Shape <= 2 and Distance >= -2 and <= 2) and then do the function for the remainder. – TheLionKing Jan 13 '16 at 19:28
  • @MooseBoys How would I tell the DLL to initialize the table? Would I create an "Initialization" function and then call it first in the main program? – TheLionKing Jan 13 '16 at 19:29
  • @TheLionKing Never mind, I thought you were talking about the cost to allocate the memory; it sounds like the real cost is in the LUT generation, in which case storing it as a separate file that you load at run-time is the best approach. – MooseBoys Jan 13 '16 at 19:33
  • 2
    I've seen recently a post about the same problem. The suggested solution was to create a separate object file out of a binary file, but use an assembler to create the object file instead of a compiler. Then link the main program against that object file. – Oleg Andriyanov Jan 13 '16 at 19:34
  • 1
    @TheLionKing, how many values will you need at any specific point of time? You maybe better of to store the data in the DB - SQLite for example, and just query the appropriate table. – Igor Jan 13 '16 at 19:41
  • @Igor Actually, that's a thought as well. Once I find the Shape for the data, all I would need is that one row of Distances, so potentially just 4001 floats at one time. – TheLionKing Jan 13 '16 at 19:47
  • 1
    @TheLionKing, Exactly my point. And since SQLite is fully embeddable , all you will need is just provide a DB with the binary. Just get the source and include it in the project. And grab the command-line tool to create a DB - you will just need 1 table. Then you are done. You may also look at some other embeddable DB engine. – Igor Jan 13 '16 at 19:51
  • @BoPersson Although not the answer to the question, the final solution was a binary file as suggested by Bo. Upload time for 8 million floats is probably less than two seconds. Thanks, Bo! – TheLionKing Jan 13 '16 at 22:16

1 Answers1

0

The compiler is probably trying to optimize out this data and that takes much more memory. GCC is able to compile a test program with this large of an array, but is taking ~2.5GiB of ram to do so. Visual Studio is likely doing the same thing, so /Zm1000 isn't actually enough.

There are alternative methods of storing blobs of data in binaries, but none of them are cross platform. With Visual Studio, the way of doing it is with resource files. With the GNU toolchain, objcopy is used. This sticks blobs into binary's the data segment, then the C++ code is linked against the symbols marking your binary blob. This avoids having the compiler trying to mess with large amounts of data. See this post for how to actually do it: https://stackoverflow.com/a/17441467/5790674.

Community
  • 1
  • 1
C. MacLean
  • 1
  • 1