Storing Enormous Static Variables in C++

Question

I have a string of information that is roughly 17 kb long. My program will not generate this string or read it into a buffer - the data is already initialized, I want it to be compiled as is from within my code, like you would a static variable. Moreover, I'd much prefer it is within my executable, and not stored within a project file. I've never before encountered such an issue, what is the best way to go around this? Should I include as resource, or literally copy and paste the enormous stream of data into a variable? What would you recommend?

Forgot to mention, am using VisualStudio C++ 2015 if that matters

Stick it in another cpp file an put an extern in a header so you can access it. That way a decent build environment won't keep rebuilding it. — user4581301, Feb 22 '17 at 06:59
That requires it to be stored in another file, I don't want the risk of files being misplaced and would prefer the ability to embed directly within the .exe — DudeDudeDude, Feb 22 '17 at 07:00
@DudeDudeDude In another **`.cpp`** file, which means it would still be compiled in and omitting it would give you linking errors. — Angew is no longer proud of SO, Feb 22 '17 at 07:01
@DudeDudeDude, you should probably consult some C++ tutorial to learn more about the code compilation process. — ForceBru, Feb 22 '17 at 07:02
I'm facepalming hard here, sorry man, evidently I need sleep, because yes that sounds correct, the code will compile with the included .cpp. Many apologies. Would you recommend pasting it into a variable or some sort of buffer in the .cpp, or is there a better way to reference such a large amount of data from an extern? I'm worried copy and pasting the stream could cause unintended errors — DudeDudeDude, Feb 22 '17 at 07:04
One big raw string literal should do it for you. It'll be `const` so no one can change it without playing silly casting games (and possibly crashing the program in thē process), and all you're doing is passing around a pointer. If you need easy access to the length or manipulation functions, use a `const std::string`. — user4581301, Feb 22 '17 at 07:32
One thing to be aware of is that Visual Studio is terrible compiling large static literal initializers of structs. We have a file containing only a large static initlialization of a struct, about 2M of data. GCC and Clang handle this easily Visual Studio takes a few minutes to compile it. If you switch optimization on it takes even longer!!! Visual Studio also exhibits an exponential slow down as the size get bigger. Note this is true up to and including VS 2017 RC4 — goneskiing, Feb 22 '17 at 11:56

score 3 · Answer 1 · edited May 23 '17 at 12:09

The GNU linker ld has the ability to directly include custom data as the .data section of an object file:

 ld -r -b binary -o example.o example.txt

The resulting example.o file has symbols defined to access start and end of the embedded data (just look at the file with objdump to see how they're named).

Now I don't know whether the linker coming with Visual Studio has a similar ability, but I guess you could use the GNU linker either via mingw or also via cygwin (since the generated object file won't reference the standard lib you won't need the emulation lib that comes with cygwin).The resulting object file apparently can just be added to your sources like a regular source file.

Of course this manual workflow isn't too good if the data changes often...

Alternatively you can write a simple program which puts the contents of the file in a C string, like:

unsigned char const * const data = { 
  0x12, 0x34, 0x56 };

Of course there's already such a program (xdd) but I don't know whether it's available to you. One potential issue is that you could reach the limit for the length of string literals that way. To get around that you could try a (multidimensional) char array.

(When writing this answer I found this blog post very helpful.)

Storing Enormous Static Variables in C++

1 Answers1