1

I'm thinking about using an embedded db4o database for storing calculation results of a simulation tool. The simulation results can be quite large (up to some GBs for a single run, so the practical size of the database might probably be in the range from 10 GB to 100 GB).

As far as I've understood, db4o stores the whole database in a single file. While I've understood that db4o supports databases up to 254 GB, I'm still worrying that I might run into problems with file systems that dislike large files (The application I'm working on will run on a big variety of architectures, so I cannot really predict which file systems will be in place...). So, is there any best practice that helps me avoiding huge files while still keeping the benefits of an embedded database?

Edit: I just found this post dealing with (really) large amounts of data in db4o in general. However, it does not go into details concerning how they achieve this without stressing the file system...

Community
  • 1
  • 1
bigge
  • 1,488
  • 15
  • 27
  • Is there any reason why you don't want to rely on a conventional database? – varocarbas Jul 22 '13 at 16:15
  • Yes, the most important reason is that the admins of the (HPC) systems I'll be running the software on won't allow me to run a database server on their machine. Computations on these machines can typically only be started as batch jobs (via a job scheduler) which further complicates the use of a real database server... – bigge Jul 22 '13 at 18:12
  • OK. I was just feeling curious about it. I don't have too much experience on this front (when the reliance on files becomes impossible, I move to (conventional) DBs) and thus am not in a position to help. – varocarbas Jul 22 '13 at 18:22

1 Answers1

1

The way to go on very large amount of data without create too large file is to use multiples containers. You should cut your data model into multiples containers (aka files).

Each file will contains some part of your model. Then by querying the good container, you should be able to retrieve yours objects; The joins (if need) must be done by hands.

If you have so few class in your model that it don't make sense to cut it, then I'm not sure db4o is way you need. Maybe a simple file with very tiny serializations should best fit yours needs.

Galigator
  • 8,957
  • 2
  • 25
  • 39
  • Well, this is what I suspected, but I was hoping that there is a more elegant approach... Still, thanks for the answer :) – bigge Jul 27 '13 at 07:57