Mongod - Insufficient free space for journal files after being hacked

Question

I got hacked and my entire production database was deleted. Now when I try to start mongod, I get the error:

 Insufficient free space for journal files
 Please make at least 3379MB available in /var/lib/mongo/journal or use --smallfiles

Im on an AWS EC2 instance and have lots of space. What can I try?

I checked /etc/mongod.conf and there is a setting there:

storage:
    dbPath: /var/lib/mongo

However, from the logs, mongod is looking in /data/db. When I then create /data/db I get the error above about insufficient space...

EDIT Ran df -kh /var/lib/mongo which returns

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  4.5G  3.3G  58% /

EDIT

Ok i increased space, that helped but now Im getting yet another error:

Failed to unlink socket file /tmp/mongodb-27017.sock errno:1 Operation not permitted

I have tried the solutions here 'Failed to unlink socket file" error in MongoDB 3.0. That is, deleting the file, changing the permissions, but everytime I start mongod, the file reappears and the same error. When I run ls -al on that file I see:

srwx------ 1 mongod mongod 0 Jun  1 18:09 /tmp/mongodb-27017.sock

[Journal files](https://docs.mongodb.com/manual/core/journaling/) are an important part of the database engine; you need to have enough space for them. — Vince Bowdren, Jun 01 '17 at 15:53
I actually do have lots of space. What happened was I got hacked and my ENTIRE production database deleted. I tried to secure the database and now Im getting this error... — Mark, Jun 01 '17 at 15:57
@robertklep returning Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7.8G 4.5G 3.3G 58% / — Mark, Jun 01 '17 at 17:15
@Mark there's 3.3G free on that partition, so it's too small. Check if `/data` or `/data/db` is not actually a symlink to `/var/`. Still though, the error you're getting explicitly mentions `/var/lib/mongo/journal`. — robertklep, Jun 01 '17 at 17:16
Yep, the needed 3379MB is 3.379G which is more than the 3.3G available. — JohnnyHK, Jun 01 '17 at 17:22
@robertklep ill try increasing my disk space on the EC2 instance and will update question — Mark, Jun 01 '17 at 17:22
I dont know how so much is suddenly used up and coincidental it should happen now after having the database hacked — Mark, Jun 01 '17 at 17:24

score 2 · Accepted Answer · answered Jun 02 '17 at 12:58

2

Increasing disk space on my EC2 instance solved the problem.

answered Jun 02 '17 at 12:58

Mark

4,428
14
60
116

Glad you fixed your problem. Just curious. How did you get around the lock permission issue ? – s7vr Jun 02 '17 at 13:10
@Veeram deleting /data/db/mongod.lock fixed it for me – Mark Jun 08 '17 at 02:41
I am experiencing a similar problem, which appeared out of nowhere. On minute my web app was working, and then it stopped. Did your issue have to do with space or hacking? – MadPhysicist May 10 '18 at 03:51
1

@MadPhysicist, mine also randomly stopped working and it turned out to be a disk space issue. Start debugging by looking at the actual MongoDB log file. That pointed it out for me. – rockstardev Aug 06 '18 at 16:46

Mongod - Insufficient free space for journal files after being hacked

1 Answers1