0

I have a large Mongo database (5M documents). I edit the database from an offline application, so I store the database on my local computer. However, I want to be able to maintain an online copy of the database, so that my website can access it.

How can I update the online copy regularly, without having to upload multiple GBs of data every time?

Is there some way to "track changes" and upload only the diff, like in Git?

Inkbug
  • 1,682
  • 1
  • 10
  • 26
  • what is exact issue what i understand - the db have 2 copy on online and one off line - you have updated of line one and want to update online one ? Question do the online db get update from app ? if not then it is possible straight forward but if it get update then . – sandeep rawat Sep 30 '21 at 13:41
  • Can't you store the commands you used on your offline db, and then apply them on the online db, through a script running on SSH for instance ? Or even better upload a file with all the commands you ran on your offline base, to your server and then execute them with a cron job, or a bash script ? (The only requirement would be for your bases to have the same start point, and same state, when you execute the script) – Youri Sep 30 '21 at 13:48
  • @Youri that sounds like a good option. How can one code that? – Inkbug Sep 30 '21 at 13:49
  • 1
    I think there can be just one _online_ copy of the database (for example in the cloud) - which can be accessed from a web app, or a command-line tool like `mongosh` or desktop GUI based tool like Compass or even a desktop custom app. – prasad_ Sep 30 '21 at 13:56
  • @prasad_ Well it could be true, it all depends on what Inkbug wants to do. But you're answering another question, classical case of XY problem i guess https://en.wikipedia.org/wiki/XY_problem :) – Youri Sep 30 '21 at 13:59

1 Answers1

1

Following up on my comment:

Can't you store the commands you used on your offline db, and then apply them on the online db, through a script running on SSH for instance ? Or even better upload a file with all the commands you ran on your offline base, to your server and then execute them with a cron job, or a bash script ? (The only requirement would be for your bases to have the same start point, and same state, when you execute the script)

I would recommend to store all the queries you execute on your offline base, to do this you have many options, I can think about the following : You can set the profiling level to log all your queries. (Here is a more detailed thread on the matter: MongoDB logging all queries)

Then you would have to extract then somehow (grep ?), or store them directly in another file on the fly, when they are executed.

For the uploading of the script, it depends on what you would like to use, but i suppose you would need to do it during low usage hours, and you could automate the task with a CRON job, and an SSH tunnel.

I guess it all depends on your constraints (security, downtime, etc..)

Youri
  • 442
  • 5
  • 15