6

I have a MongoDB client in three EC2 instances and I have created a replica set. Last time I had a problem, of space constraint which stopped my mongod process, thereby halting the application and now in an instance couple of days back, some of my tables were gone from database, so I set logging and all to my database just to catch if anything like that happens again. In a fresh incident this morning I was unable to login to my system and that's when I found out that whole database was empty. I checked other SO question like this which suggest setting up a TTL.Which I haven't done at all.
Now how do I debug this situation and do a proper root cause analysis? I can't even find anything in my debug logs as well. The tables just vanished. How do I set up proper logging mechanism and how do I ensure that all my tables are never ever deleted again?

Community
  • 1
  • 1
Saras Arya
  • 3,022
  • 8
  • 41
  • 71
  • this is weird - almost identical question here - http://stackoverflow.com/questions/41519988/mongodb-collections-unexpected-deletion-drop/41520214#41520214 – Alex Jan 07 '17 at 10:53
  • Yeah... I am soon going into production. can't have something like this happening. I read your answer. Couldn't it be like mongo removed your database because of lack of memory? – Saras Arya Jan 07 '17 at 12:08
  • 1
    No, it wouldn't just randomly delete collections - that would be a really poor thing to do (which it doesn't) - definitely accidental deletion – Alex Jan 07 '17 at 12:10
  • How do I setup logs so that I can catch all commands executed on a replica set? So if it happens next time I would know what command caused it – Saras Arya Jan 07 '17 at 12:18
  • maybe this http://stackoverflow.com/questions/15204341/mongodb-logging-all-queries – Alex Jan 07 '17 at 12:32
  • 2
    What specific version of MongoDB are you running? Is your replica set properly secured (see: [MongoDB Security Checklist](https://docs.mongodb.com/manual/administration/security-checklist/)? If you haven't enabled authentication and locked down access to your replica set, one possibility is that someone is remotely deleting data. If you definitely have authentication configured and enabled, I would review your logs to try to determine when the databases were dropped. Start by grepping for `dropDatabase` (log lines may vary depending on your version and config settings). – Stennie Jan 11 '17 at 11:04
  • @Stennie I just did that Thank you very much for your help. Now my config is secured. Sorry for late reply. – Saras Arya Jan 19 '17 at 13:36

1 Answers1

4

Today I got a mail from Amazon that I was probably running an unsecured version of MongoDB and that may have caused this issue. So who ever is facing this issue please go through the Security Checklist Provided by MongoDB. There are some points that are absolutely necessary in there.
1. Enable Access Control and Enforce Authentication
2. Encrypt Communication
3. Limit Network Exposure

These three are the core and depending upon how many people access your database you can Configure Role-Based Access Control.

These are all the things I have done. Before this incident I had not taken security that seriously but after I was hit by it. I made sure I have all the necessary precautions in place. Hope this helps someone.

Saras Arya
  • 3,022
  • 8
  • 41
  • 71