MongoDB killed by linux

Question

My mongoDB always receives a KILL signal when it is a few hours without being accessed and I do not know the reason.

I'm using:
S.O: Ubuntu 16.04 with 512MB ram.
Running node 7.5.0 in pm2.
MongoDB version 3.4.1

status mongod:

mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
Active: failed (Result: signal) since Fri 2017-06-02 06:25:06 BRT; 9h ago
 Docs: https://docs.mongodb.org/manual
Process: 1334 ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf (code=killed, signal=KILL)
Main PID: 1334 (code=killed, signal=KILL)

May 29 19:12:16 admin systemd[1]: Started High-performance, schema-free document-oriented database.
Jun 02 06:25:06 admin systemd[1]: mongod.service: Main process exited, code=killed, status=9/KILL
Jun 02 06:25:06 admin systemd[1]: mongod.service: Unit entered failed state.
Jun 02 06:25:06 admin systemd[1]: mongod.service: Failed with result 'signal'.

Mongod.service:

[Unit]
Description=High-performance, schema-free document-oriented database
After=network.target
Documentation=https://docs.mongodb.org/manual

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongod --quiet --config /etc/mongod.conf
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

[Install]
WantedBy=multi-user.target

etc/mongod.conf:

storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

net:
  port: 27017
  bindIp: 127.0.0.1

processManagement:
  fork: false

journalctl:

Jun 01 22:32:02 admin pm2[23634]: [PM2] Spawning PM2 daemon with pm2_home=/home/administrator/.pm2
Jun 01 22:32:02 admin pm2[23634]: [PM2] PM2 Successfully daemonized
Jun 01 22:32:02 admin pm2[23634]: [PM2] Stopping PM2...
Jun 01 22:32:02 admin pm2[23634]: [PM2][WARN] No process found
Jun 01 22:32:02 admin pm2[23634]: [PM2] All processes have been stopped and deleted
Jun 01 22:32:02 admin pm2[23634]: [PM2] PM2 stopped
Jun 01 22:32:06 admin pm2[23649]: pm2 launched in no-daemon mode (you can add DEBUG="*" env variable to get more messages)
Jun 01 22:32:06 admin pm2[23649]: 2017-06-01 22:32:06: Launching in no daemon mode
Jun 01 22:32:06 admin pm2[23649]: 2017-06-01 22:32:06: [PM2] Resurrecting
Jun 01 22:32:06 admin pm2[23649]: 2017-06-01 22:32:06: [PM2] Restoring processes located in /home/administrator/.pm2/dump.pm2
Jun 01 22:32:06 admin pm2[23649]: 2017-06-01 22:32:06: [Watch] Start watching 0
Jun 01 22:32:06 admin pm2[23649]: 2017-06-01 22:32:06: Starting execution sequence in -fork mode- for app name:MY_NODE id:1
Jun 01 22:32:06 admin pm2[23649]: 2017-06-01 22:32:06: App name:MY_NODE id:1 online
Jun 01 22:32:06 admin pm2[23649]: 2017-06-01 22:32:06: [PM2] Process /home/servers/nodejs/server_MY_NODE/server.js restored
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07: ┌─────────────┬────┬──────┬───────┬────────┬─────────┬────────┬─────┬───────────┬──────────┐
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07: │ App name    │ id │ mode │ pid   │ status │ restart │ uptime │ cpu │ mem       │ watching │
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07: ├─────────────┼────┼──────┼───────┼────────┼─────────┼────────┼─────┼───────────┼──────────┤
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07: │   MY_NODE   │ 1  │ fork │ 23665 │ online │ 0       │ 0s     │ 24% │ 23.9 MB   │ disabled │
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07: └─────────────┴────┴──────┴───────┴────────┴─────────┴────────┴─────┴───────────┴──────────┘
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07:  Use `pm2 show <id|name>` to get more details about an app
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07: [--no-daemon] Continue to stream logs
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07: [--no-daemon] Exit on target PM2 exit pid=23649
Jun 01 22:32:07 admin pm2[23649]: 2017-06-01 22:32:07: [STREAMING] Now streaming realtime logs for [all] processes
Jun 01 22:32:07 admin pm2[23649]: 22:32:07 1|MY_NODE | WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
Jun 01 22:32:07 admin pm2[23649]: 22:32:07 1|MY_NODE | WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
Jun 01 22:32:09 admin pm2[23649]: 22:32:09 1|MY_NODE | Express Server in 10555
Jun 01 22:32:09 admin pm2[23649]: 22:32:09 1|MY_NODE | Database! connected in mongodb://localhost/MY_MONGO
Jun 02 06:25:05 admin pm2[23649]: 2017-06-02 06:25:05: App [MY_NODE] with id [1] and pid [23665], exited with code [0] via signal [SIGKILL]
Jun 02 06:25:05 admin pm2[23649]: 2017-06-02 06:25:05: Starting execution sequence in -fork mode- for app name:MY_NODE id:1
Jun 02 06:25:05 admin pm2[23649]: 06:25:05 PM2        | App [MY_NODE] with id [1] and pid [23665], exited with code [0] via signal [SIGKILL]
Jun 02 06:25:05 admin pm2[23649]: 06:25:05 PM2        | Starting execution sequence in -fork mode- for app name:MY_NODE id:1
Jun 02 06:25:05 admin pm2[23649]: 2017-06-02 06:25:05: App name:MY_NODE id:1 online
Jun 02 06:25:05 admin pm2[23649]: 06:25:05 PM2        | App name:MY_NODE id:1 online
Jun 02 06:25:06 admin pm2[23649]: 06:25:06 1|MY_NODE | WARNING: NODE_APP_INSTANCE value of '0' did not match any instance config file names.
Jun 02 06:25:06 admin pm2[23649]: 06:25:06 1|MY_NODE | WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
Jun 02 06:25:08 admin pm2[23649]: 06:25:08 1|MY_NODE | Express Server in 10555
Jun 02 06:25:08 admin pm2[23649]: 06:25:08 1|MY_NODE | Database! disconnected from mongodb://localhost/MY_MONGO
Jun 02 06:25:08 admin pm2[23649]: 06:25:08 1|MY_NODE | Database! Error in connection mongodb://localhost/MY_MONGO

Who can help me, I'll be grateful

thanks

How is your machine doing on RAM? Do you have swap space available? Is is possible that the memory creep of the mongod process is causing an OOM event / to be terminated by oom-killer? What is the output of `free`? May be encountering [this](https://jira.mongodb.org/browse/SERVER-22000). — Matt Clark, Jun 02 '17 at 19:25
the server have 250MB avaliabled and the swap is not actived. — Armando, Jun 02 '17 at 20:17
If you do not have any swap space allocation, then I **highly** assume that you are running out of physical memory, and the process is probably being killed by [oom-killer](https://stackoverflow.com/questions/624857/finding-which-process-was-killed-by-linux-oom-killer). — Matt Clark, Jun 02 '17 at 20:18
@MattClark, is this, i exec command "dmesg" and show message: "Killed process 1334 (mongod) total-vm:760892kB, anon-rss:30944kB, file-rss:11812kB" — Armando, Jun 02 '17 at 22:53
Checked dmesg Out of memory: Kill process 16279 (mongod) score 14 or sacrifice child Killed process 16279 (mongod) total-vm:4245156kB, anon-rss:629216kB, file-rss:0kB Server has 32GB RAM, almost free and 16GB swap - all free. — ulikus, Sep 13 '18 at 13:32

score 7 · Answer 1 · answered Nov 24 '20 at 14:39

7

According to the Mongo-DB documentation, Ulimit can affect the mongo service. I fixed the same problem by running: ulimit -a to verify this was the problem.

If the -n flag is less than 64,000 you have to change it: sudo ulimit -n 64000

Then restart mongo: sudo systemctl restart mongod

And make sure it is running now: systemctl status mongod

answered Nov 24 '20 at 14:39

c10ud

196
3
12

You should also be able to set the limits in the *.service* file itself. Nowadays, if you installed mongo by it's official install script those should already be set up. – Aros Oct 04 '21 at 19:09

MongoDB killed by linux

1 Answers1