0

I need to save a large 3D array of integers into a file, and load it again in C++. It is 256*256*256 = 16777216 integers.

What is the best way to save this and load it again? I am mostly interested in a quick load time.

genpfault
  • 51,148
  • 11
  • 85
  • 139
hansdam
  • 127
  • 3
  • 11
  • You should be able to load from the file using multiple threads (as any number of readers are fine). Storing them in the same array has to be done carefully (so as to not interfere with each other). I would try and save the data in such a way as to make multiple threads feasible (i.e. a line break format). – twain249 Apr 11 '12 at 20:41
  • 1
    @twain249 I'd say the bottleneck is the disk, adding to seek times by moving it back and forth within the file won't make it better by much... Just sayin... – littleadv Apr 11 '12 at 20:44
  • Maybe see question [How to compress a buffer with zlib](http://stackoverflow.com/q/4538586/1084416) – Peter Wood Apr 11 '12 at 20:54

2 Answers2

8

If the array is allocated in contiguous memory (i.e.: you don't allocate each dimension separately) - you can just dump the whole memory block to file. It takes as much as it takes, but that would be the least overhead (i.e.: call binary write on the whole chunk of data).

If you're saving on one system and loading on another, you might have issues with data representation, in this case you'd probably want to serialize the array and save each value in a controlled matter.

littleadv
  • 20,100
  • 2
  • 36
  • 50
1

You may be interested in Boost.Serialization, particularly if you (1) want the ability to easily store such data on disk, (2) want a coherent way to save more complex objects, and (3) want a solution that's portable.

pmcs
  • 1,123
  • 10
  • 16