MongoDB databases disappeared

Question

My entire mongo database seems to be gone (this is for my thesis which is due in a month so kind of desperate here).

I'm running some code fetching tweets (it's been working for a month without any edits) and writing to mongo using pymongo and I get this error:

Traceback (most recent call last):
   File "linguistic_feats.py", line 100, in <module>
write_all_user_tweets(conn, dataset)
   File "linguistic_feats.py", line 89, in write_all_user_tweets
[get_all_tweets(user, conn) for user in parse]
   File "linguistic_feats.py", line 89, in <listcomp>
[get_all_tweets(user, conn) for user in parse]
   File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/collect.py", line 58, in get_all_tweets
[conn.write_user_collection(tweet) for tweet in  all_tweets]
   File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/collect.py", line 58, in <listcomp>
[conn.write_user_collection(tweet) for tweet in  all_tweets]
   File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/db.py", line 49, in write_user_collection
table.update({'_id': tweet['id_str']}, {'$setOnInsert': tweet}, upsert = True)
   File "/usr/local/lib/python3.5/site-packages/pymongo/collection.py", line 2230, in update
with self._socket_for_writes() as sock_info:
   File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
   File "/usr/local/lib/python3.5/site-packages/pymongo/mongo_client.py", line 712, in _get_socket
server = self._get_topology().select_server(selector)
   File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 141, in select_server
address))
   File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 117, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 61] Connection refused

I tried to access the mongo console but receive:

MongoDB shell version: 3.2.0
connecting to: test
2016-06-24T03:02:05.622+0200 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017 after 5000 milliseconds, giving up.
2016-06-24T03:02:05.623+0200 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:224:14
@(connect):1:6

exception: connect failed

Then I called mongod manually, which allowed me to access the console but now instead of having multiple show dbs simply gives:

$ mongo
MongoDB shell version: 3.2.0
connecting to: test
Server has startup warnings:
2016-06-24T03:07:43.638+0200 I CONTROL  [initandlisten]
2016-06-24T03:07:43.638+0200 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
> use thesis 
> show dbs
local  0.000GB
> show collections
>

Log file:

2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:73125][473:0x70000020a000], log-server: /usr/local/var/mongodb/journal: opendir: Too many open files
2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74536][473:0x70000020a000], log-server: dirlist journal prefix WiredTigerPreplog: Too many open files
2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74568][473:0x70000020a000], log-server: log pre-alloc server error: Too many open files
 2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74588][473:0x70000020a000], log-server: log server error: Too many open files
 2016-06-24T01:17:51.078+0200 W FTDC     [ftdc] Uncaught exception in 'FileNotOpen Failed to open interim file /usr/local/var/mongodb/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
 2016-06-24T01:18:14.124+0200 E STORAGE  [thread2] WiredTiger (24) [1466723894:124125][473:0x700000393000], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: fopen: Too many open files
 2016-06-24T01:18:14.127+0200 E STORAGE  [thread2] WiredTiger (24) [1466723894:126992][473:0x700000393000], checkpoint-server: checkpoint server error: Too many open files
 2016-06-24T01:18:14.127+0200 E STORAGE  [thread2] WiredTiger (-31804) [1466723894:127084][473:0x700000393000], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-06-24T01:18:14.127+0200 I -        [thread2] Fatal Assertion 28558
 2016-06-24T01:18:14.127+0200 I -        [thread2] 

***aborting after fassert() failure


2016-06-24T01:18:14.163+0200 F -        [thread2] Got signal: 6 (Abort trap: 6).

0x10788d449 0x10788cddf 0x7fff8deb952a 0x7fff94e7ba26 0x7fff9ce4e6e7 0x10782d53a 0x1076679a1 0x10800e381 0x10800e4c9 0x10800ea74 0x107f9ac5d 0x7fff98d5199d 0x7fff98d5191a 0x7fff98d4f351
----- BEGIN BACKTRACE -----
{
  "backtrace": [
    {
      "b": "106FA5000",
  "o": "8E8449"
},
{
  "b": "106FA5000",
  "o": "8E7DDF"
},
{
  "b": "7FFF8DEB7000",
  "o": "252A"
},
{
  "b": "7FFF94E79000",
  "o": "2A26"
},
{
  "b": "7FFF9CDF0000",
  "o": "5E6E7"
},
{
  "b": "106FA5000",
  "o": "88853A"
},
{
  "b": "106FA5000",
  "o": "6C29A1"
},
{
  "b": "106FA5000",
  "o": "1069381"
},
{
  "b": "106FA5000",
  "o": "10694C9"
},
{
  "b": "106FA5000",
  "o": "1069A74"
},
{
  "b": "106FA5000",
  "o": "FF5C5D"
},
{
  "b": "7FFF98D4E000",
  "o": "399D"
},
{
  "b": "7FFF98D4E000",
  "o": "391A"
},
{
  "b": "7FFF98D4E000",
  "o": "1351"
}
  ],
  "processInfo": {
"mongodbVersion": "3.2.0",
"gitVersion": "45d947729a0315accb6d4f15a6b06be6d9c19fe7",
"compiledModules": [

],
"uname": {
  "sysname": "Darwin",
  "release": "15.5.0",
  "version": "Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8\/RELEASE_X86_64",
  "machine": "x86_64"
},
"somap": [
  {
    "path": "\/usr\/local\/opt\/mongodb\/bin\/mongod",
    "machType": 2,
    "b": "106FA5000",
    "buildId": "A553ECE2755B341A850927E8DEE071E3"
  },
  {
    "path": "\/usr\/lib\/libSystem.B.dylib",
    "machType": 6,
    "b": "7FFF92227000",
    "buildId": "C5D09FE1CC70383EAC2718602F2EDEC4"
  },
  {
    "path": "\/usr\/lib\/libc++.1.dylib",
    "machType": 6,
    "b": "7FFF9112A000",
    "buildId": "8FC3D139805534989AC56467CB7F4D14"
  },
  {
    "path": "\/usr\/lib\/system\/libcache.dylib",
    "machType": 6,
    "b": "7FFF9ACA9000",
    "buildId": "9548AAE92AB735259ECEA2A7C4688447"
  },
  {
    "path": "\/usr\/lib\/system\/libcommonCrypto.dylib",
    "machType": 6,
    "b": "7FFF8B009000",
    "buildId": "9373226134B43914B7A290A81A182DBA"
  },
  {
    "path": "\/usr\/lib\/system\/libcompiler_rt.dylib",
    "machType": 6,
    "b": "7FFF9DD3D000",
    "buildId": "A13ECF69F59F38AE86097B731450FBCD"
  },
  {
    "path": "\/usr\/lib\/system\/libcopyfile.dylib",
    "machType": 6,
    "b": "7FFF8E288000",
    "buildId": "A48637BCF3F234F2BB684C65FD012832"
  },
  {
    "path": "\/usr\/lib\/system\/libcorecrypto.dylib",
    "machType": 6,
    "b": "7FFF9F808000",
    "buildId": "B5C05FD7A540345A87BF8E41848A3C17"
  },
  {
    "path": "\/usr\/lib\/system\/libdispatch.dylib",
    "machType": 6,
    "b": "7FFF8BEA7000",
    "buildId": "C749985761A53D7DA5EA65DCC8C3DF92"
  },
  {
    "path": "\/usr\/lib\/system\/libdyld.dylib",
    "machType": 6,
    "b": "7FFF8B915000",
    "buildId": "CC088C2AD40733E7A6B6B06E0D4AD999"
  },
  {
    "path": "\/usr\/lib\/system\/libkeymgr.dylib",
    "machType": 6,
    "b": "7FFF9B28B000",
    "buildId": "8371CE545FDD3CE9B3DFE98C761B6FE0"
  },
  {
    "path": "\/usr\/lib\/system\/liblaunch.dylib",
    "machType": 6,
    "b": "7FFF92532000",
    "buildId": "834ED60551143641AA4DECF31B801C50"
  },
  {
    "path": "\/usr\/lib\/system\/libmacho.dylib",
    "machType": 6,
    "b": "7FFF92499000",
    "buildId": "318264FA58F139D882851F6254EE410E"
  },
  {
    "path": "\/usr\/lib\/system\/libquarantine.dylib",
    "machType": 6,
    "b": "7FFF8C27F000",
    "buildId": "0F4169F00C843A25B3AEE47B3586D908"
  },
  {
    "path": "\/usr\/lib\/system\/libremovefile.dylib",
    "machType": 6,
    "b": "7FFF8DEC0000",
    "buildId": "552EF39E14D7363E90594565AC2F894E"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_asl.dylib",
    "machType": 6,
    "b": "7FFF8AFB5000",
    "buildId": "41F8E11F1BD03F1DBA3AAA1577ED98A9"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_blocks.dylib",
    "machType": 6,
    "b": "7FFF91908000",
    "buildId": "1244D9D5F6AA35BBB30786851C24B8E5"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_c.dylib",
    "machType": 6,
    "b": "7FFF9CDF0000",
    "buildId": "B552D565B7983B9BAE63F623B42A5F01"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_configuration.dylib",
    "machType": 6,
    "b": "7FFF8E876000",
    "buildId": "3DEB7DF9680437E1BC830166882FF0FF"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_coreservices.dylib",
    "machType": 6,
    "b": "7FFF8B8FE000",
    "buildId": "1B3F5AFCFFCD3ECB8B9A5538366FB20D"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_coretls.dylib",
    "machType": 6,
    "b": "7FFF9CDBC000",
    "buildId": "C90DAE384082381CA1852A6A8B677628"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_dnssd.dylib",
    "machType": 6,
    "b": "7FFF92CDD000",
    "buildId": "4D10E12B59B5386F82DA326F18028F0A"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_info.dylib",
    "machType": 6,
    "b": "7FFF98C85000",
    "buildId": "FAA9226D64DE3769A6D86CABA4B7FF4D"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_kernel.dylib",
    "machType": 6,
    "b": "7FFF98CDF000",
    "buildId": "78E54D59D2B03F549A4A0A68D671F253"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_m.dylib",
    "machType": 6,
    "b": "7FFF9F7D7000",
    "buildId": "08E1A4B264483DFEA58CACC7335BE7E4"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_malloc.dylib",
    "machType": 6,
    "b": "7FFF94E79000",
    "buildId": "5748E8B2F81C34C68B13456213127678"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_network.dylib",
    "machType": 6,
    "b": "7FFF9A5E8000",
    "buildId": "B52DAB7392DC3DA7B9F4B899D66445C1"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_networkextension.dylib",
    "machType": 6,
    "b": "7FFF97679000",
    "buildId": "66095DC7653938F295EE458F15F6D014"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_notify.dylib",
    "machType": 6,
    "b": "7FFF9942C000",
    "buildId": "D48BDE340F7E34CAA0FFC578E39987CC"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_platform.dylib",
    "machType": 6,
    "b": "7FFF8DEB7000",
    "buildId": "29A905EF67773C3382B06C3A88C4BA15"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "machType": 6,
    "b": "7FFF98D4E000",
    "buildId": "3DD1EF4C1D1B3ABF8CC6B3B1CEEE9559"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_sandbox.dylib",
    "machType": 6,
    "b": "7FFF921C2000",
    "buildId": "150A9D3DF69E32F78C7B8E72CAAFF7E4"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_secinit.dylib",
    "machType": 6,
    "b": "7FFF95103000",
    "buildId": "32B1A8C6DC843F4FB8CE9A52B47C3E6B"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_trace.dylib",
    "machType": 6,
    "b": "7FFF993CB000",
    "buildId": "F00E92E4DBDA3749B5B30C3FBBABA1CB"
  },
  {
    "path": "\/usr\/lib\/system\/libunc.dylib",
    "machType": 6,
    "b": "7FFF9ECDF000",
    "buildId": "DDB1E947C77533B8B46163E5EB698F0E"
  },
  {
    "path": "\/usr\/lib\/system\/libunwind.dylib",
    "machType": 6,
    "b": "7FFF8B125000",
    "buildId": "F6EB48E54D12359AAB54C937FBBE9043"
  },
  {
    "path": "\/usr\/lib\/system\/libxpc.dylib",
    "machType": 6,
    "b": "7FFFA10B6000",
        "buildId": "54D1328E054E3DAA89E2375722F9D18F"
      },
      {
        "path": "\/usr\/lib\/libobjc.A.dylib",
        "machType": 6,
        "b": "7FFF9E13F000",
        "buildId": "7489D2D61EFD3414B18D2AECCCC90286"
      },
      {
        "path": "\/usr\/lib\/libauto.dylib",
        "machType": 6,
        "b": "7FFF9B6C1000",
        "buildId": "999E610F41FC32A3ADCA5EC049B65DFB"
      },
      {
        "path": "\/usr\/lib\/libc++abi.dylib",
        "machType": 6,
        "b": "7FFF8E88B000",
        "buildId": "DCCC81773D0935BC97842A04FEC4C71B"
      },
      {
        "path": "\/usr\/lib\/libDiagnosticMessagesClient.dylib",
        "machType": 6,
        "b": "7FFF96E9C000",
        "buildId": "4243B6B421E9355B9C5A95A216233B96"
      }
    ]
  }
}

[UPDATE]

Now the daemon won't even start anymore.

2016-06-24T10:05:32.986+0200 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-06-24T10:05:32.986+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=9G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-06-24T10:05:33.226+0200 E STORAGE  [initandlisten] WiredTiger (-31803) [1466755533:226568][6493:0x7fff75e89000], txn-recover: Recovery failed: WT_NOTFOUND: item not found
2016-06-24T10:05:33.248+0200 I -        [initandlisten] Assertion: 28595:-31803: WT_NOTFOUND: item not found
2016-06-24T10:05:33.249+0200 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31803: WT_NOTFOUND: item not found, terminating
2016-06-24T10:05:33.249+0200 I CONTROL  [initandlisten] dbexit:  rc: 100

You've reached the open file limit of your operating system. Edit /etc/security/limits.conf to increase limits. This may guide you : http://stackoverflow.com/questions/20901518/ubuntu-too-many-open-files — Mustafa Genç, Jun 24 '16 at 01:37
@MustafaGenç Thanks so much for the quick answer, but why would I not be able to access the database via console? Or how could I get access to the data already written (without attempting new writes)? — ZeerakW, Jun 24 '16 at 02:41
@MustafaGenç I tried to update the limits - and while it does not complain about limits anymore I still cannot access the previously written databases — ZeerakW, Jun 24 '16 at 03:07
@profesor79 No it's not but when I run the daemon I can't access my databases. Actually now I can't even start the daemon anymore. — ZeerakW, Jun 24 '16 at 08:06
this can happen recovery after crash. there exists bug report about this. https://jira.mongodb.org/browse/SERVER-19774 . But no clean solution is suggested. — Mustafa Genç, Jun 24 '16 at 08:51

score 12 · Accepted Answer · answered Jun 25 '16 at 06:24

12

In your manual restart , it looks like you are running the mongodb with all defaults (the dbpath is /data/db, whereas the original ulimit error was logging /usr/local/var/mongodb).

Which path is supposed to be correct?

If you ran it manually with the wrong dbpath you would see an empty db.

Then if you didn't cleanly stop this process (e.g kill - 9), that could have corrupted this instance (hopefully on an incorrect dbpath ).

First, take a copy of the /usr/local/var/mongodb path.

Then try restarting the mongod process with the correct dbpath, user, permissions and ulimit settings (the earlier /etc/security/limits.conf suggestion was likely not useful on a Mac).

If none of this helps, then maybe try upgrade first (3.2.0 is quite old).

answered Jun 25 '16 at 06:24

jimoleary

1,645
13
11

1

Hey, I'm not entirely sure what you mean by which path is supposed to be correct? One path is the logging path and the other path is the database path? Also tried updating to 3.2.7 (latest one on home-brew it seems) to no avail. – ZeerakW Jun 25 '16 at 08:09
1

You're a bleeding genius!! The correct DBpath was /usr/local/var/mongodb and the DB recovered itself just perfectly after updating the ulimit! – ZeerakW Jun 25 '16 at 08:14
2

Your welcome! Might be worth backing up occasionally going forward ;) – jimoleary Jun 25 '16 at 09:52
@jimoleary I was facing the same problem and your given solution worked for me and can you shed some more light on how mongodb started showing me collection when i pointed the --dbpath to /usr/local/var/mongodb. In my understanding if no db path is given , mongo stores the data in data/db directory but how come pointing dapath to /usr/local/var/mongodb showed me docs which were under data/db directory earlier. Does this mean when i kill the mongodb server , mongo makes sure that data of it's instance is copied over to /usr/local/var/mongodb from data/db directory. – sam Feb 29 '20 at 15:30

MongoDB databases disappeared

1 Answers1