1

We are using MongoDB as an intermediate storage for an application that allows the user uploading downloading video files. We are using GridFS API from a Java applications , as is very convenient for the case (We found it more appropriate, faster and reliable than storing the files in a table in a relation database).

Once the videos have been processed from the DB (and stored into physical files) we can remove them, but we have the problem that the new space is not just reallocated and is instead "used" without any util data. We have tried to repair the database, as suggested in posts like Auto compact the deleted space in mongodb? but this had the database down for few days! , which is not ideal as it needs to be running 24/7. (We have come accoss this recently when the DB was without free space).

I am not very knowledgeable in this topic and so would like to get opinions for a solution you know/use that would be efficient: Allow the storage in blocks and be easy to reallocate memory once the blocks/chunks are not needed. Some options are:

1) Have two Mongo DB: from time to time export and import data(all except the tables used by GridFS that contain the videos) from one db to the other. First db can be dropped and space defragmented again. This seems a bit complex and not good if it needs to be done frequently as we don't have much total space for the DB.

2) Store them in a relational database (for a table without relation and these special characteristics does not seem ideal but works if other solutions don't)

3) ...

If it serves, the application is deployed in a J2EE infrastructure.

Thanks.

Community
  • 1
  • 1
kandan
  • 715
  • 2
  • 12
  • 39
  • According to [this question](http://stackoverflow.com/questions/26942680/when-data-deleted-from-mysql-what-happens-to-space-it-took-up), when something is deleted in a database, the space is free to be reused by the next insert command. Is your database actually running out of space and preventing further data to be stored? – Question Marks Dec 04 '14 at 22:11
  • yes, deleting some documents liberated space. However, it still displays the size is full which seems confusing (mongo does not defragment). Main problem is that one column stores the block's encrypted binary data, and the size is always different. We found that very occasionally for some users saving some very big blocks was not working ok as it couldn't find a block of a similar size to allocate for that block, if that makes sense so we suspect it is not a definite solution in our case. Probably something we are doing wrong, looking for similar cases as well. – kandan Dec 04 '14 at 22:46
  • 1
    are you running this as a replica set? If so, resyncing the secondary will compact all the space and then you can fail-over to it and do the same with the other node(s). – Asya Kamsky Dec 05 '14 at 00:19
  • No we are not. We need to setup a some structure to backup the main db and allow this ; I think this sounds good (similar to the option 1 I mentioned with the idea of replication). Will check with some sys admin as don't know whether have a script that at some point copies/exports the non-video data from one db to the other or some process that backups the data as it is being inserted in the main node (master/slave) and then switches the nodes. thks. – kandan Dec 05 '14 at 00:55

0 Answers0