29

i'm trying to find the best solution to create scalable storage for big files. File size can vary from 1-2 megabytes and up to 500-600 gigabytes.

I have found some information about Hadoop and it's HDFS, but it looks a little bit complicated, because i don't need any Map/Reduce jobs and many other features. Now i'm thinking to use MongoDB and it's GridFS as file storage solution.

And now the questions:

  1. What will happen with gridfs when i try to write few files concurrently. Will there be any lock for read/write operations? (I will use it only as file storage)
  2. Will files from gridfs be cached in ram and how it will affect read-write perfomance?
  3. Maybe there are some other solutions that can solve my problem more efficiently?

Thanks.

Community
  • 1
  • 1
cmd
  • 515
  • 3
  • 9
  • 19

3 Answers3

21

I can only answer for MongoDB here, I will not pretend I know much about HDFS and other such technologies.

The GridFs implementation is totally client side within the driver itself. This means there is no special loading or understanding of the context of file serving within MongoDB itself, effectively MongoDB itself does not even understand they are files ( http://docs.mongodb.org/manual/applications/gridfs/ ).

This means that querying for any part of the files or chunks collection will result in the same process as it would for any other query, whereby it loads the data it needs into your working set ( http://en.wikipedia.org/wiki/Working_set ) which represents a set of data (or all loaded data at that time) required by MongoDB within a given time frame to maintain optimal performance. It does this by paging it into RAM (well technically the OS does).

Another point to take into consideration is that this is driver implemented. This means that the specification can vary, however, I don't think it does. All drivers will allow you to query for a set of documents from the files collection which only houses the files meta data allowing you to later serve the file itself from the chunks collection with a single query.

However that is not the important thing, you want to serve the file itself, including its data; this means that you will be loading the files collection and its subsequent chunks collection into your working set.

With that in mind we have already hit the first snag:

Will files from gridfs be cached in ram and how it will affect read-write perfomance?

The read performance of small files could be awesome, directly from RAM; the writes would be just as good.

For larger files, not so. Most computers will not have 600 GB of RAM and it is likely, quite normal in fact, to house a 600 GB partition of a single file on a single mongod instance. This creates a problem since that file, in order to be served, needs to fit into your working set however it is impossibly bigger than your RAM; at this point you could have page thrashing ( http://en.wikipedia.org/wiki/Thrashing_%28computer_science%29 ) whereby the server is just page faulting 24/7 trying to load the file. The writes here are no better as well.

The only way around this is to starting putting a single file across many shards :\.

Note: one more thing to consider is that the default average size of a chunks "chunk" is 256KB, so that's a lot of documents for a 600GB file. This setting is manipulatable in most drivers.

What will happen with gridfs when i try to write few files concurrently. Will there be any lock for read/write operations? (I will use it only as file storage)

GridFS, being only a specification uses the same locks as on any other collection, both read and write locks on a database level (2.2+) or on a global level (pre-2.2). The two do interfere with each other as well, i.e. how can you ensure a consistent read of a document that is being written to?

That being said the possibility for contention exists based on your scenario specifics, traffic, number of concurrent writes/reads and many other things we have no idea about.

Maybe there are some other solutions that can solve my problem more efficiently?

I personally have found that S3 (as @mluggy said) in reduced redundancy format works best storing a mere portion of meta data about the file within MongoDB, much like using GridFS but without the chunks collection, let S3 handle all that distribution, backup and other stuff for you.

Hopefully I have been clear, hope it helps.

Edit: Unlike what I accidently said, MongoDB does not have a collection level lock, it is a database level lock.

Manquer
  • 7,390
  • 8
  • 42
  • 69
Sammaye
  • 43,242
  • 7
  • 104
  • 146
  • I _think_ the global lock was changed? (https://blog.serverdensity.com/goodbye-global-lock-mongodb-2-0-vs-2-2/) – Jeff Apr 07 '14 at 23:27
  • 1
    @Jeff this is an old answer, I could update it if people are still using it? – Sammaye Apr 08 '14 at 07:08
  • @Jeff oh hang on I actually say database level lock, where do I say global? – Sammaye Apr 08 '14 at 07:08
  • I may have posted the wrong article. And I can't find the other one... I was using this answer as I'm trying to determine what would be better for file storage, Mongo or a FS – Jeff Apr 08 '14 at 13:39
  • @Jeff oh FS easily for big files, for smaller....hmmm, still a tough one even now, for avatars I have gone for storing into the user document but for videos I went for S3... – Sammaye Apr 08 '14 at 13:50
  • FYI, Just got an email from 10gen: `Finally, MongoDB 2.6 lays the foundation for massive improvements to concurrency in MongoDB 2.8, including document-level locking.` – Jeff Apr 08 '14 at 16:32
  • @Jeff I just got it too :D hope that does actually come in 2.8, fingers crossed – Sammaye Apr 08 '14 at 16:33
  • Well with some drivers you could end up with the whole file in RAM which is obviously bad but in NodeJS you it will be perfectly fine as you can use streams so it won't crash your server but sure it would take lot of resources of it. – antoineg Dec 07 '16 at 21:24
  • The database and the application are two separate layers. However, across languages the database implements the same base – Sammaye Dec 07 '16 at 21:30
6

Have you considered saving meta data onto MongoDB and writing actual files to Amazon S3? Both have excellent drivers and the latter is highly redundant, cloud/cdn-ready file storage. I would give it a shot.

mluggy
  • 83
  • 5
  • 1
    Concur, with S3. I saw this Google Groups groups post, https://groups.google.com/forum/?fromgroups=#!topic/mongoose-orm/G85Q2QaA1QI, explored GridFS and then came back to this point of view. – prototype Feb 23 '13 at 02:23
4

I'll start by answering the first two:

  1. There is a write lock when writing in to GridFS, yes. No lock for reads.
  2. The files wont be cached in memory when you query them, but their metadata will.

GridFS may not be the best solution for your problem. Write locks can become something of a pain when you're dealing with this type of situation, particularly for huge files. There are other databases out there that may solve this problem for you. HDFS is a good choice, but as you say, it is very complicated. I would recommend considering a storage mechanism like Riak or Amazon's S3. They're more oriented around being storage for files, and don't end up with major drawbacks. S3 and Riak both have excellent admin facilities, and can handle huge files. Though with Riak, last I knew, you had to do some file chunking to store files over 100mb. Despite that, it generally is a best practice to do some level of chunking for huge file sizes. There are a lot of bad things that can happen when transferring files in to DBs- From network time outs, to buffer overflows, etc. Either way, your solution is going to require a fair amount of tuning for massive file sizes.

Christopher WJ Rueber
  • 2,141
  • 15
  • 22
  • There is a rad lock for reading from gridfs, the files can be cached in memory according to the OS LRU if the computers memory is big enough for such a working set. – Sammaye Feb 22 '13 at 19:36
  • Chris, thanks for your answer. Few more questions about HDFS. Are there any locks for reading/writing in this distributed file system that can be as painful as locks in GridFS? And what about limitations for NameNode(only one or multiple instaces). Maybe i'll try to experiment with it – cmd Feb 22 '13 at 20:39
  • @Sammaye The "working set" is equivalent to the index. On GridFS it only loads that, not all the files. If it did it would be next to useless. – Christopher WJ Rueber Feb 23 '13 at 00:18
  • @cmd I don't actually know all that much about HDFS. I don't believe it has any read/write locks, because it actually is treated like a file system. Unfortunately, HDFS had a high enough barrier to entry that I just haven't messed around with it much. So take anything about HDFS that I say with a grain of salt. Riak and S3 on the other hand- Those I've used quite regularly, and definitely think they're both viable solutions for the type of problem that you're solving. – Christopher WJ Rueber Feb 23 '13 at 00:19
  • Gridfs is no different to making a normal query in MongoDB as such the data is paged in as a normal query. There is no special resolution of gridfs data on the server-side, in fact the server has no idea about gridfs, it is completely driver implemented. – Sammaye Feb 23 '13 at 00:26
  • The working set is not the index, the working set is the total data required by the process in a given period of time to maintain optimal performance – Sammaye Feb 23 '13 at 00:31
  • @Sammaye That is only half correct- Yes, it loads the whole working set, but that's the files metadata. Which is what I am calling the index. The "chunks" collection (where the files live) is not loaded unless you directly call it. – Christopher WJ Rueber Feb 23 '13 at 00:31
  • Yes but if you get the file contents then that is querying it...so to answer the question; yes files can reside in ram – Sammaye Feb 23 '13 at 00:32
  • @Sammaye Correct, but that's just a single file. Which is chunked up in to many documents in the collection. Which each individually are put back together to make the whole, but each query is only for one individual document, keeping the working set extremely small. Here's a link for more details: http://docs.mongodb.org/manual/applications/gridfs/ – Christopher WJ Rueber Feb 23 '13 at 00:33
  • No the working set is all data loaded from MongoDB (the process) http://en.wikipedia.org/wiki/Working_set and also no, the chunks query is a single query for all document chunks of that file in most drivers – Sammaye Feb 23 '13 at 00:34
  • I'd recommend checking the mongo manual for further details, Sammaye: http://docs.mongodb.org/manual/faq/storage/#what-is-the-working-set – Christopher WJ Rueber Feb 23 '13 at 00:36
  • Whereby it says: "If you run a query that requires MongoDB to scan every document in a collection, the working set includes every active document in memory." Edit: actually this line "Often this is a subset of the total data size, but the specific size of the working set depends on actual moment-to-moment use of the database." – Sammaye Feb 23 '13 at 00:37
  • The exception for when documents of a query are not loaded into the working set is on index only cursors, whereby all data is garnished solely from the index, in which case the index is the only part included into the working set for that query. – Sammaye Feb 23 '13 at 00:48
  • @Sammaye wouldn't you have an index-only cursor when retrieving the chunks from GridFS though? – Anentropic Jun 28 '13 at 11:56
  • @Anentropic No....why would the file data be stored in the index?? That would just be crazy and extremely resource heavy...far too much for even my 65GB memory server – Sammaye Jun 28 '13 at 11:58
  • @Sammaye agreed, I misunderstood what was meant by an index-only cursor – Anentropic Jun 28 '13 at 12:26