I am making a video sharing site. What is the best way to store lots of big video files, so that it will be fast and performance efficient while accessing them?
A colleague said that storing as files on hard-disk folder is not going to be efficient as it will be very slow on many read/write operations.
Someone else suggested storing them in MySQL blobs.
Generally storing large files in a database is not considered good practice. See this question and those linked from it for further info.
Rather than simply storing on disk, you may wish to consider using a service such as Rackspace Cloud Files or Amazon S3 if your budget allows.
Use your ID key in the database to sort files into folder.
Take the first number, put that in first directory, second number in next.. and so on:
ex:
18
/videos/1/8/MyvideoFile
1892
/videos/1/8/9/2/MyVideoFile
Alternatively you can also sort them on various servers or NAS clusters by storing a server number in your DB.
Hope that helps.
Storing in a mysql DB will be slower than storing as binary files on a HDD for sure. You could store copies of the files on multiple machines if one machine can't handle the load, and use an off the shelf load balancer to manage it.
A solution that would require a small upfront time investment on your part but that would save you a lot of hassle in the long run... is using a cloud storage service like:
http://aws.amazon.com/s3/ or http://www.rackspacecloud.com/cloud_hosting_products/files
It's scalable and your servers would be able to concentrate on running your web app.
Videos are too big to be stored efficiently in MySQL. Use the file system. Come up with a reasonable way to sort the video files into separate directories, so that you don't end up with thousands of files in a single directory. For example, all files that start with 'a' go into directory 'a', etc.
If your site gets really big, it might make sense to offload your videos to a CDN, such as Amazon Cloudfront.
So for example you have a file of size 50 GB , you can write it on 50 disks in parallel , 1 GB part on each disk , so it you take just 1 second to write and similarly then it should take 1 second to read from 50 disks. Consider file systems like HDFS.
