2

I am designing a solution to store many large files. From what I have read on stackoverflow, database systems such as Mongodb are the best way to store a large number of files (source: Scalable File Storage). Why? Surely I would be better off simply storing the files in the file system on a scaleable system such as like Amazon and giving each file an ID.

To try to find an answer I have looked at:

So far I appreciate that:

  1. Storing on the file system would make server migration a lot harder.
  2. There are limits as to the number of files and folders that can be kept in a directory.

Neither of these two points seems to convince me that it is worth the extra work of programming files to be kept in a database. It also feels to me that it would increase the risk of a catastrophic corruption of the database if I store large files in it rather than storing directly in the file system. I am currently thinking of storing the files in an Amazon s3 solution is this. Can you please let me know if am crazy and/or if I have overlooked anything?

Community
  • 1
  • 1
Brett
  • 3,296
  • 5
  • 29
  • 45
  • 1
    I'm not sure exactly what you're doing, but if there is money involved, I would use a 3rd party to ensure that they will be stored safely. In most cases however, I would store them on the file system. The thing is, you need to come up with a good database schema to organize the files and all their info. Another question you have to ask yourself is, are the files supposed to be protected? If so, you need to take extra steps to ensure they cannot easily be accessed by URL manipulation. – half-fast Apr 27 '13 at 11:16
  • 1
    The best way to store large files is on your server's filesystem, or on a cdn, not in a database - only store references to files in a database – Mark Baker Apr 27 '13 at 11:16
  • 1
    I would use S3. The only condition I would consider developing my own system is if there were compliance requirements that would rule out using S3. On a small scale it would be very expensive to build a system to build a system with the same performance/reliability as s3. – datasage Apr 27 '13 at 11:21

2 Answers2

0

If you would like to store your files on self hosted server then you can choose MongoDB. MongoDB is the best option to store the large file data over the self hosted server. But you want to keep out of from the maintenance of self hosted server then you can choose the Amazon S3.

Ganesh Kanawade
  • 381
  • 1
  • 4
0

Database is meant for data not files! I have come across many situation where many prefer to store it to database, be it mangodb or others. Logically that's not worth.

Your question "is it worth of programming effort?"; seriously no if you are doing once in a while. It takes a lot of effort to put things in database. However if you are a developer working on it frequently once you are get used to you, you will do it even if its not worth to do so :)

I vote for you to go for file system storage for files and not database. And for difficulties of no. of files, you will find a way to resolve it for sure.

PC.
  • 481
  • 7
  • 23
  • There are benefits to database storage as well which make it worth it, in fact many cdns store file data in a chunk format stored in a database – Sammaye Apr 27 '13 at 11:51
  • @Sammaye, completely agree with you. But there is difference between CDNs which are full fledged system with much better resources than our normal hosting servers. – PC. Apr 27 '13 at 12:00
  • True though, it is something to bare in mind before database hosting is automatically cast off for no real explanation :) – Sammaye Apr 27 '13 at 12:31