1

I'm unable to stop, start or restart mongodb and so I have stopped the AWS server altogether.

Here's the log file from /var/log/mongodb/mongod.log. Any idea why?

{"t":{"$date":"2021-04-18T11:23:59.814+00:00"},"s":"I",  "c":"CONTROL",  "id":20698,   "ctx":"main","msg":"***** SERVER RESTARTED *****"}
{"t":{"$date":"2021-04-18T11:23:59.817+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2021-04-18T11:23:59.825+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2021-04-18T11:23:59.826+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2021-04-18T11:23:59.826+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":3617,"port":27017,"dbPath":"/var/lib/mongo","architecture":"64-bit","host":"ip-XXX-AA-BB-YYY.us-east-2.compute.internal"}}
{"t":{"$date":"2021-04-18T11:23:59.826+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.5","gitVersion":"ff5cb77101b052fa02da43b8538093486cf9b3f7","openSSLVersion":"OpenSSL 1.0.2k-fips  26 Jan 2017","modules":[],"allocator":"tcmalloc","environment":{"distmod":"amazon2","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2021-04-18T11:23:59.826+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Amazon Linux release 2 (Karoo)","version":"Kernel 4.14.203-156.332.amzn2.x86_64"}}}
{"t":{"$date":"2021-04-18T11:23:59.826+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017},"processManagement":{"fork":true,"pidFilePath":"/var/run/mongodb/mongod.pid","timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/var/lib/mongo","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"path":"/var/log/mongodb/mongod.log"}}}}
{"t":{"$date":"2021-04-18T11:23:59.827+00:00"},"s":"I",  "c":"STORAGE",  "id":22270,   "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/var/lib/mongo","storageEngine":"wiredTiger"}}
{"t":{"$date":"2021-04-18T11:23:59.827+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=256M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2021-04-18T11:24:00.504+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":13,"message":"[1618745040:504678][3617:0x7f4abfab2280], wiredtiger_open: __posix_open_file, 808: /var/lib/mongo/WiredTiger.turtle: handle-open: open: Permission denied"}}
{"t":{"$date":"2021-04-18T11:24:00.504+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":13,"message":"[1618745040:504970][3617:0x7f4abfab2280], wiredtiger_open: __posix_open_file, 808: /var/lib/mongo/WiredTiger.turtle: handle-open: open: Permission denied"}}
{"t":{"$date":"2021-04-18T11:24:00.505+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":13,"message":"[1618745040:505157][3617:0x7f4abfab2280], wiredtiger_open: __posix_open_file, 808: /var/lib/mongo/WiredTiger.turtle: handle-open: open: Permission denied"}}
{"t":{"$date":"2021-04-18T11:24:00.505+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2021-04-18T11:24:00.505+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"13: Permission denied"}}
{"t":{"$date":"2021-04-18T11:24:00.505+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":947}}
{"t":{"$date":"2021-04-18T11:24:00.505+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

Here's what happens when I try to start or see the status:

[root@ip-XXX-AA-BB-YYY ~]# sudo service mongod start
Redirecting to /bin/systemctl start mongod.service
Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
[root@ip-XXX-AA-BB-YYY ~]# sudo systemctl status mongod
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2021-04-18 11:24:00 UTC; 41s ago
     Docs: https://docs.mongodb.org/manual
  Process: 3613 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=14)
  Process: 3610 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 3607 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 3606 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)

Apr 18 11:23:59 ip-XXX-AA-BB-YYY.us-east-2.compute.internal systemd[1]: Starting MongoDB Database Server...
Apr 18 11:23:59 ip-XXX-AA-BB-YYY.us-east-2.compute.internal mongod[3613]: about to fork child process, waiting until server is ready for connections.
Apr 18 11:23:59 ip-XXX-AA-BB-YYY.us-east-2.compute.internal mongod[3613]: forked process: 3617
Apr 18 11:24:00 ip-XXX-AA-BB-YYY.us-east-2.compute.internal systemd[1]: mongod.service: control process exited, code=exited status=14
Apr 18 11:24:00 ip-XXX-AA-BB-YYY.us-east-2.compute.internal systemd[1]: Failed to start MongoDB Database Server.
Apr 18 11:24:00 ip-XXX-AA-BB-YYY.us-east-2.compute.internal systemd[1]: Unit mongod.service entered failed state.
Apr 18 11:24:00 ip-XXX-AA-BB-YYY.us-east-2.compute.internal systemd[1]: mongod.service failed.
Elaine Byene
  • 3,868
  • 12
  • 50
  • 96
  • maybe the solutions in [this question](https://stackoverflow.com/questions/53478123/mongodb-can-not-start-because-of-wiredtiger-turtle-permissions) might help you. if not, you might have better luck wiping the VM and reinstalling mongo as instructured [here](https://docs.mongodb.com/manual/tutorial/install-mongodb-on-amazon/) – Dĵ ΝιΓΞΗΛψΚ Apr 18 '21 at 15:30

2 Answers2

0

Check this mongodb Jira ticket, its has exact resemblance of your scenario https://jira.mongodb.org/browse/SERVER-38247?attachmentViewMode=list

Moreover, for Exit-code 14 error --> this code problem means that your mongodb SOCK dont belong to mongodb user. So here is a quick fix : Go to

/tmp and find mongodb sock

cd /tmp; ls -l *.sock

  1. Change the owner of that .sock file, it used to named as mongodb-27017.sock, but if you find it in a different name, its ok.

    chown mongodb:mongodb mongodb-27017.sock

The user and group should be same which are using by mongodb data and log path or entire service was running.

Kaushik Das
  • 405
  • 1
  • 5
  • 12
0

If you did a log cleanup on the server some files might be missing or created with root access wich blocks mongodb user from using it

To fix this (again, this is only on the log cleanup case):

sudo mkdir -p /var/lib/mongodb/journal
sudo chown -R mongodb:mongodb /var/lib/mongodb/journal
# Be sure only mongodb user can run the files
# permissions on /var/lib/mongodb/journal/ files should be 600
sudo mkdir -p /var/log/mongodb/
sudo chown -R mongodb:mongodb /var/log/mongodb/

You might also want to apply Kaushik Das's solution before starting the server.

Be sure to kill mongodb service and not start it before you put mongodb group on the log files otherwise the process will create them as root

savageGoat
  • 1,389
  • 1
  • 3
  • 10