15

I need to store very large (more than 512Mb) binary files into a NoSQL database. What particular NoSQL database implementation allows that?

surfmuggle
  • 5,527
  • 7
  • 48
  • 77
Edward83
  • 6,664
  • 14
  • 74
  • 102
  • 1
    You should clarify what you're trying to do randomly accessing binary files. Random access in a binary file sounds suspiciously like a database. I'm not sure you want to put a database in your database ;) –  Nov 24 '10 at 01:07
  • 1
    http://stackoverflow.com/questions/12894542/mongodb-gridfs-file-sizes-huge-for-relatively-small-file According to this mongo should not be good for large files, it collapses by 2gb summary storage place ... Try to find a key-value database for that, riak and redis are good, but larger files than 50mb are not recommended by that databases. :S Maybe mcdb is good for that: http://stackoverflow.com/questions/9724319/cdb-constant-key-value-store-for-large-files-hundreds-of-gb – inf3rno Jul 03 '13 at 12:52
  • related: http://stackoverflow.com/q/2890452/462865 – Amir Ali Akbari Jan 16 '14 at 11:40

3 Answers3

8

No experience, but Mongos GridFS is the only thing I have heard of that is specifically for storing files

Matt Briggs
  • 41,224
  • 16
  • 95
  • 126
  • 1
    +1, i made some tests 2 days ago. it works for MongoDB, but i want to try find something else for compare;) – Edward83 Nov 24 '10 at 12:57
7

If you're going to store binary files in a non-relational database, you could look into using Luwak. You could also look into using Hadoop's HDFS to store your files in a distributed fashion. Neither of these solutions will do much for you until you have three or more servers to run them on and a fast network backbone to support the file replication.

Without knowing more about the rationale for storing large amounts of binary data in a database, it doesn't make much sense. Your filesystem does an amazing job of storing binary files with searchable metadata. If you're just looking to catalog your movies and their metadata, there are better solutions. If you're trying to get distributed and redundant storage for your movies, there are better solutions out there.

6

There's a super lean NoSQL storage mechanism with excellent support for random access. In fact, there are many competing implementations of a common, standardized interface with years of testing from hundreds of millions of users around the world.

Look no further than fopen. It's web scale!

Ben Karel
  • 4,591
  • 2
  • 26
  • 25