4

I'm trying to determine the best way to store large numbers of small .mat files, around 9000 objects with sizes ranging from 2k to 100k, for a total of around half a gig.

The typical use case is that I only need to pull a small number (say 10) of the files from disk at a time.

What I've tried:

Method 1: If I save each file individually, I get performance problems (very slow save times and system sluggishness for some time after) as Windows 7 has difficulty handling so may files in a folder (And I think my SSD is having a rough time of it, too). However, the end result is fine, I can load what I need very quickly. This is using '-v6' save.

Method 2: If I save all of the files in one .mat file and then load just the variables I need, access is very slow (loading takes around three quarters of the time it takes to load the whole file, with small variation depending on the ordering of the save). This is using '-v6' save, too.

I know I could split the files up into many folders but it seems like such a nasty hack (and won't fix the SSD's dislike of writing many small files), is there a better way?

Edit: The objects are consist mainly of a numeric matrix of double data and an accompanying vector of uint32 identifiers, plus a bunch of small identifying properties (char and numeric).

Shai
  • 111,146
  • 38
  • 238
  • 371
MatlabSorter
  • 1,290
  • 1
  • 11
  • 19
  • related question: [MATLAB: Saving several variables to "-v7.3" (HDF5) .mat-files seems to be faster when using the "-append" flag. How come?](http://stackoverflow.com/questions/4949939/matlab-saving-several-variables-to-v7-3-hdf5-mat-files-seems-to-be-faster) – Amro Aug 17 '11 at 00:36

3 Answers3

2

Five ideas to consider:

  1. Try storing in an HDF5 object - take a look at http://www.mathworks.com/help/techdoc/ref/hdf5.html - you may find that this solves all of your problems. It will also be compatible with many other systems (e.g. Python, Java, R).
  2. A variation on your method #2 is to store them in one or more files, but to turn off compression.
  3. Different datatypes: It may also be the case that you have some objects that compress or decompress inexplicably poorly. I have had such issues with either cell arrays or struct arrays. I eventually found a way around it, but it's been awhile & I can't remember how to reproduce this particular problem. The solution was to use a different data structure.
  4. @SB proposed a database. If all else fails, try that. I don't like building external dependencies and additional interfaces, but it should work (the primary problem is that if the DB starts to groan or corrupts your data, then you're back at square 1). For this purpose consider SQLite, which doesn't require a separate server/client framework. There is an interface available on Matlab Central: http://www.mathworks.com/matlabcentral/linkexchange/links/1549-matlab-sqlite
  5. (New) Considering that the objects are less than 1GB, it may be easier to just copy the entire set to a RAM disk and then access through that. Just remember to copy from the RAM disk if anything is saved (or wrap save to save objects in two places).

Update: The OP has mentioned custom objects. There are two methods to consider for serializing these:

  1. Two serialization program from Matlab Central: http://www.mathworks.com/matlabcentral/fileexchange/29457 - which was inspired by: http://www.mathworks.com/matlabcentral/fileexchange/12063-serialize
  2. Google's Protocol Buffers. Take a look here: http://code.google.com/p/protobuf-matlab/
Iterator
  • 20,250
  • 12
  • 75
  • 111
  • Thanks Iterator. I am already saving them without compression (V6) as I've found the more recent formats tend to be slower to access and often result in larger filesizes (even with compression). I'll take a look at HDF5. – MatlabSorter Aug 15 '11 at 22:44
  • Matlab's implementation of HDF5 doesn't seem to support custom objects, either. – MatlabSorter Aug 15 '11 at 23:04
  • Can you describe the objects? It might be the case that they can be transformed and accessed more quickly. There was a strange thing about -V7 and -V7.3 in the compression; for everything under 2GB, I tended to use -V7. I think most recent versions of Matlab use HDF5, but some (all?) versions have a known bug in their usage that caused extremely slowing writing. That's why I tended to use older formats for everything small. – Iterator Aug 16 '11 at 02:30
1

Try storing them as blobs in a database.

I would also try the multiple folders method as well - it might perform better than you think. It might also help with organization of the files if that's something you need.

NG.
  • 22,560
  • 5
  • 55
  • 61
  • Thanks SB. I can't find a way to serialize custom objects in Matlab, so I would still need to save to disk before populating the database. – MatlabSorter Aug 15 '11 at 22:47
0

The solution I have come up with is to save object arrays of around 100 of the objects each. These files tend to be 5-6 meg so loading is not prohibitive and access is just a matter of loading the right array(s) and then subsetting them to the desired entry(ies). This compromise avoids writing too many small files, still allows for fast access of single objects and avoids any extra database or serialization overhead.

MatlabSorter
  • 1,290
  • 1
  • 11
  • 19