0

I am using aws cloudformation to build mongo cluster on CentOS7 and mongodb V3.0.7. The build script will need to bounce each mongod instances after adding logins to enable authentication. Sometimes, after shutdwon mongod (I tried different ways to shutdown by using using db.serverShutdown() or kill -2 pid or systemctl restart mongod.service), it always complains unclean shutdown, could not bring mongod instance up again. The journaling is enabled.

Here is mongod.log

2015-12-19T18:53:38.734+0000 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2015-12-19T18:57:41.207+0000 I INDEX    [conn7] allocating new ns file       /db/mongodb/27017/data/admin.ns, filling with zeroes...
2015-12-19T18:57:41.913+0000 I STORAGE  [FileAllocator] allocating new datafile /db/mongodb/27017/data/admin.0, filling with zeroes...
2015-12-19T18:57:41.916+0000 I STORAGE  [FileAllocator] done allocating datafile /db/mongodb/27017/data/admin.0, size: 16MB,  took 0.002 secs
2015-12-19T18:57:41.922+0000 I INDEX    [conn7] build index on: admin.system.users properties: { v: 1, unique: true, key: { user: 1, db: 1 }, name: "user_1_db_1", ns: "admin.system.users" }
2015-12-19T18:57:41.922+0000 I INDEX    [conn7]          building index using bulk method
2015-12-19T18:57:41.922+0000 I INDEX    [conn7] build index done.  scanned 0 total records. 0 secs
2015-12-19T18:57:41.922+0000 I COMMAND  [conn7] command admin.$cmd command: createUser { createUser: "mongodbm", pwd: "xxx", digestPassword: false, roles: [ "root" ] } keyUpdates:0 writeConflicts:0 numYields:0 reslen:37 locks:{ Global: { acquireCount: { r: 8, w: 8 } }, MMAPV1Journal: { acquireCount: { w: 11 } }, Database: { acquireCount: { w: 4, W: 4 } }, Collection: { acquireCount: { W: 1 } }, Metadata: { acquireCount: { W: 7 } }, oplog: { acquireCount: { w: 4 } } } 744ms
2015-12-19T18:58:22.245+0000 I REPL     [ReplicationExecutor] transition to SECONDARY
2015-12-19T18:58:22.245+0000 I COMMAND  [conn9] terminating, shutdown command received
2015-12-19T18:58:22.245+0000 I REPL     [conn9] Stopping replication applier threads
2015-12-19T18:58:22.776+0000 I CONTROL  [conn9] now exiting
2015-12-19T18:58:22.776+0000 I NETWORK  [conn9] shutdown: going to close listening sockets...
2015-12-19T18:58:22.776+0000 I NETWORK  [conn9] closing listening socket: 6
2015-12-19T18:58:22.776+0000 I NETWORK  [conn9] closing listening socket: 7
2015-12-19T18:58:22.776+0000 I NETWORK  [conn9] removing socket file: /db/mongodb/27017/etc/mongodb-27017.sock
2015-12-19T18:58:22.776+0000 I NETWORK  [conn9] shutdown: going to flush diaglog...
2015-12-19T18:58:22.776+0000 I NETWORK  [conn9] shutdown: going to close sockets...
2015-12-19T18:58:22.776+0000 I STORAGE  [conn9] shutdown: waiting for fs preallocator...
2015-12-19T18:58:22.776+0000 I STORAGE  [conn9] shutdown: final commit...
2015-12-19T18:58:22.776+0000 I JOURNAL  [conn9] journalCleanup...
2015-12-19T18:58:22.776+0000 I JOURNAL  [conn9] removeJournalFiles
2015-12-19T18:58:22.779+0000 I JOURNAL  [conn9] Terminating durability thread ...
2015-12-19T18:58:22.879+0000 I JOURNAL  [journal writer] Journal writer thread stopped
2015-12-19T18:58:22.879+0000 I JOURNAL  [durability] Durability thread stopped
2015-12-19T18:58:22.879+0000 I STORAGE  [conn9] shutdown: closing all files...
2015-12-19T18:58:22.880+0000 I STORAGE  [conn9] closeAllFiles() finished
2015-12-19T18:58:22.880+0000 I STORAGE  [conn9] shutdown: removing fs lock...
2015-12-19T18:58:22.880+0000 I CONTROL  [conn9] dbexit:  rc: 0
2015-12-19T18:58:52.911+0000 I CONTROL  ***** SERVER RESTARTED *****
2015-12-19T18:58:52.965+0000 W -        [initandlisten] Detected unclean shutdown - /db/mongodb/27017/data/mongod.lock is not empty.
2015-12-19T18:58:52.971+0000 I STORAGE  [initandlisten] **************
old lock file: /db/mongodb/27017/data/mongod.lock.  probably means unclean shutdown,
but there are no journal files to recover.
this is likely human error or filesystem corruption.
please make sure that your journal directory is mounted.
found 2 dbs.
see: http://dochub.mongodb.org/core/repair for more information
*************

When I check mongod.lock file, it's not there, not sure why it's complaining:

# ls -lrt  /db/mongodb/27017/data/mongod.lock
ls: cannot access /db/mongodb/27017/data/mongod.lock: No such file or directory

Here is mongod.conf file:

#mongod.conf
# where to write logging data.
systemLog:
   destination: file
   logAppend: true
   path: /db/mongodb/27017/logs/mongod.log
   quiet: true
   logRotate: reopen

# Where and how to store data.
storage:
   dbPath: /db/mongodb/27017/data
   journal:
       enabled: true

   mmapv1:
      smallFiles:  true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
   fork: true
   pidFilePath: /db/mongodb/27017/data/mongod.lock
# network interfaces
net:
   port: 27017
   unixDomainSocket:
   pathPrefix: /db/mongodb/27017/etc

replication:
   oplogSizeMB: 1
   replSetName: replset_2

sharding:
   clusterRole: "shardsvr"

security:
    authorization: enabled
    keyFile: /db/mongodb/27017/etc/keyfile

Does anyone know what's wrong ?

Jean
  • 1
  • 4
  • Possible duplicate of [Mongodb service wont start](http://stackoverflow.com/questions/18524925/mongodb-service-wont-start) – BatScream Dec 20 '15 at 08:13
  • That's not the same issue. first, why normal shutdown caused "unclean shutdown" ? the second is that there is no "mongod.lock" in /db/mongodb/27017/data directory. – Jean Dec 22 '15 at 15:23
  • @BatScream OP is right, this is not a duplicate, it's a very subtle error in what file is selected for pidFilePath. – Asya Kamsky Dec 29 '15 at 16:40

1 Answers1

0

You are telling it to create your pid file in pidFilePath as <dbpath>/mongod.lock but that's an internal file mongod uses to track whether it shut down cleanly.

So you are likely causing this yourself by making it create this file in the DB directory on startup (and of course it's gone by the time the startup fails).

Asya Kamsky
  • 41,784
  • 5
  • 109
  • 133