c++ loading large amount of data at compile time

Question

I have a C++ object which needs a huge amount of data to instantiate. For example:

  class object {
      public object() { 
          double a[] = { array with 1 million double element }; 
      /* rest of code here*/};
      private:
      /* code here*/    
  }

Now the data (i.e 1 million double numbers) is in a separate text file. The question: How can I put it after "double a[]" in an efficient way and eventually compile the code? I do not want to read the data at run time from a file. I want it compiled with the object. What can be a solution? Ideally I would like the data to sit in the separate text file as it presently resides and somehow also have an assignment like double a[] =..... above.

Is this possible? Thanks in advance!

Note: assuming a 64 bits `double` this means **8 MB** in the `.rodata` section of your object. Are you stress-testing the compiler ;) ? — Matthieu M., Nov 30 '12 at 07:34

score 10 · Answer 1 · answered Nov 30 '12 at 01:58

10

Something like:

class object
{
  public
  object(){ double a[] = { 
     #include "file.h"
  }; 
   /* rest of code here*/};
  private:
  /* code here*/    
}

The file has to be formatted correctly though - i.e. contain something like:

//file.h
23, 24, 40,
5, 1.1,

In general, you can use #include directives to paste content into files. I've seen virtual methods being pasted like that, if they were common for most derived classes. I personally don't really like this technique.

answered Nov 30 '12 at 01:58

Luchian Grigore

253,575
64
457
625

beat me by a nano second :) – Trevor Hickey Nov 30 '12 at 01:59
"I personally don't really like this technique." But still, there's no other way. – Mark Garcia Nov 30 '12 at 02:02
the data file is data each in separate line that is it looks like 1,2,3,... each of them in separate line. will this still work with such an formatting ? – user1612986 Nov 30 '12 at 02:03
@MarkGarcia You can always run a script that generates the whole source. – Luchian Grigore Nov 30 '12 at 02:03
@user1612986 then no, the syntax of array initialization has to be preserved. – Luchian Grigore Nov 30 '12 at 02:03
@LuchianGrigore Not everyone likes making scripts. – Mark Garcia Nov 30 '12 at 02:03
@MarkGarcia sometimes you do stuff you don't like to avoid duplication. – Luchian Grigore Nov 30 '12 at 02:04
@LuchianGrigore LOL. It took time to sink in me. – Mark Garcia Nov 30 '12 at 02:06
@LuchianGrigore do you mean that if the data is in separate lines it is not possible ? because 1 million data cannot be in a,b,c,...in one single line. can that be possible ? – user1612986 Nov 30 '12 at 02:10
@user1612986 it can be in separate lines, but they have to be separated by commas. – Luchian Grigore Nov 30 '12 at 02:10
1

@user1612986 actually, the example I showed you clearly show 2 separate lines. – Luchian Grigore Nov 30 '12 at 02:11
1

+1 for answer and another +1 (if I could) for "I don't like this". – Mark B Nov 30 '12 at 03:30

Yakk - Adam Nevraumont · Answer 2 · 2012-11-30T14:52:57.550

One large problem with this design is that 1 million ints on the stack will probably blow the stack. What you probably want is to put the data on the data segment, or in some kind of resource that is stores in your binary file and can be loaded at run time. If you need more than one copy of the data, duplicate it into a std::vector at run time, so you know the data is on the free store (heap). Mayhap even use a shared_ptr to a std::array to reduce the chance of needless accidental duplication(or unique_ptr to reduce the chance of reference duplication).

4mb of data is not going to play all that well is all I am saying. And locality of reference on a 4mb array to your other variables is not going to be your biggest concern.

Depending in your compiled target platform and framework, there will be ways to stuff this kind of data into a binary resource. I've never done it for a multi-meg file, but here is the visual studio help on resource files: http://msdn.microsoft.com/en-us/library/7zxb70x7%28v=vs.80%29.aspx

Note that "the data being in the code" does not make it fundamentally faster to load (other than traversing the filesystem once to find it maybe). The OS still has to load the binary, and larger binaries take more time to load, and a big array of values will take up as much room in a binary as it does in a distinct file. The real advantage is that it isn't a file that can be "misplaced" relative to your executable, but resource fork/resource file/etc methods can deal with that.

As noted in the comments below, static const data (and global data) tends to be loaded into the data segment, which is distinct from both the heap (aka free store) and stack (aka automatic store). I forget what the standard calls it. I do know that a static local variable in a function will behave differently than a static or global non-local variable with regards to initialization order (global (static or not) data gets initialized fully prior to main starting, while static local is initialized the first time the function is called, if I remember correctly).

How do you know it will be on the stack? What if you create it with `new object`? Where would the memory be then? — Luchian Grigore, Nov 30 '12 at 05:39
To resolve this problem the array a must be static (and const). `const static a[] = {....}` — SergV, Nov 30 '12 at 09:13
@SergV do you mean that when we create array as const static it gets created in the heap ? i did not know about this and learnt a new things. thanks. — user1612986, Nov 30 '12 at 12:46
@user1612986. static data will be created in specific partition of memory. It is not heap and it is not stack. Run your compiler and create assembler listing file and you will see all details. Map file from linker contain a lot of information too. — SergV, Nov 30 '12 at 13:14

score 0 · Answer 3 · edited May 23 '17 at 12:06

0

The answer of Luchian Grigore is quite correct. But compiler can have some limit on length of source code line. See for example https://stackoverflow.com/questions/10519738/source-line-length-limit

So try on your compiler. But I am afraid, more simple solution of your problem will be reading of huge data from file.

edited May 23 '17 at 12:06

Community

1
1

answered Nov 30 '12 at 03:36

SergV

1,269
8
20

c++ loading large amount of data at compile time

3 Answers3