PM2 keeps getting killed every 90 seconds on centos 8

Question

I just installed CentOS 8 and added nodejs (tried v12 & v14) And then I installed pm2 using npm install pm2@latest (so at the time of posting it uses v4.4.0). I did try an older version (v3.5.0), but it does the exact same thing.

and after pm2 got installed, i ran the command "pm2 startup"

after a restart, pm2 does start, but gets killed after 90 seconds and then restarts giving this message

"pm2 has been killed by signal, dumping process list before exit..."

First, I thought it was because of my app (the one that pm2 is supposed to manage), but i removed it from pm2, so it's practically empty, but it does the same thing

I'm seeing the same thing on RHEL 8.2, node v12.16.1. Every 90 seconds `pm2 has been killed by signal, dumping process list before exit...` Have you found any solution? I don't see anything useful in my logs. I have the same versions and setup running in another environment, but I don't see the issue there. But every 90 seconds, restart. — baronnoraz, Jul 17 '20 at 20:11
I have a similar issue. It isn't always 90 seconds, sometimes it is 60 seconds and sometimes 120 seconds. I did find that pm2 tests for memory usage every 30 seconds which seems to be related but haven't found anything yet. — Jedediah Smith, Jul 20 '20 at 22:04
Have you found anything? The same is happening for me, exactly 90 seconds every time. Whether I'm just running one API or multiple, the whole pm2 process gets restarted.. — peterdotplus, Jul 27 '20 at 14:36
Yes, please read below (3 options). Use either the second or the third for now. — Alex Laz, Aug 02 '20 at 00:13

score 13 · Answer 1 · edited Nov 04 '21 at 08:45

13

Running the following command as root worked for me:

pm2 update

edited Nov 04 '21 at 08:45

Tomerikoo

18,379
16
47
61

answered Nov 04 '21 at 08:42

Roman

167
2
9

Could you provide more details on what this command does? – Tyler2P Nov 04 '21 at 16:42
It updates pm2 itself. – Roman Nov 08 '21 at 17:16
I didn't use sudo and it worked for me (Oracle Linux 8). However, I'm wondering if this needs to be run each time the server restarts and the service begins. Has anybody encountered that behavior or was it just fixed for good at that point? – cwadrupldijjit Apr 28 '22 at 21:33

score 9 · Answer 2 · edited Apr 28 '21 at 09:26

9

I had the same issue and I tried several solutions online but none worked for me.

However, I completely removed pm2, restarted the server, and reinstalled pm2 and that does it for me.

1- Stop and remove pm2

pm2 kill
sudo npm remove pm2 -g

2- Restart the server

sudo reboot

3- Log in again, then reinstall pm2

sudo npm install -g pm2

edited Apr 28 '21 at 09:26

Hello World

2,673
7
28
60

answered Sep 29 '20 at 10:19

Shakirah

197
1
4

You might as well remove this because it has nothing to do with Ubuntu. It's a SELinux issue, which doesn't come installed with Ubuntu. – Alex Laz Sep 30 '20 at 14:20
@shakirah typo in the last step: "pm", need "pm2" – Epexa Nov 21 '20 at 05:23
This seems to solve the issue for me. Thanks :) – Hello World Apr 24 '21 at 11:52
solved for me, but I had to do: "pm2 update" too – Albanir Neves May 18 '21 at 14:24
running on ubuntu server, this also solved for me. `sudo service pm2-ubuntu status` would state that a PID file could not be read. after this reinstall this error is no longer shown – Caina Santos Jun 28 '21 at 22:28

Epexa · Answer 3 · 2021-03-30T01:29:37.423

9

I did not disable SE Linux (I think it's not safe to disable it), but the following method helped me:

Edit file: /etc/systemd/system/pm2-root.service

Add new line: Environment=PM2_PID_FILE_PATH=/run/pm2.pid
And replace: PIDFile=/root/.pm2/pm2.pid to: PIDFile=/run/pm2.pid

Versions:

CentOS 8.3.2011
Node.js 14.16.0
NPM 7.7.5
PM2 4.5.5

Original answer. Thanks Alec!

edited Mar 30 '21 at 01:29

answered Mar 30 '21 at 01:23

Epexa

265
2
8

I had the same problem in Fedora with a pm2 instantiated for a user different to root. To avoid changing permissions I've used the /tmp/ folder instead of the /run/ folder – Davide Apr 20 '21 at 10:36
Worked for us on Rocky Linux too, thanks :) – poshest Mar 11 '22 at 11:58
This worked to me too, of course using /tmp/ instead of /run/ – SleepWritten Jun 03 '22 at 19:37

score 7 · Answer 4 · answered Aug 02 '20 at 00:11

7

Later update. For those who are facing the same issues. It's an issue related to SE Linux. Known workarounds (the ones I discovered).

Disabling SE Linux (obviously, not recommended)
go to /etc/systemd/system/pm2-root.service - comment PIDFile=... (add a # in front of that line)
Audit and trace - use following commands:
```
 # dnf install policycoreutils-python-utils setroubleshoot-server -y
 # journalctl -f
```
At ths point, you should see the solution in the output (the log) it should be something like:
```
# ausearch -c 'systemd' --raw | audit2allow -M my-systemd
# semodule -i my-systemd.pp
```
You need to do the last step (ausearch... and semodule...) twice - I did it once, restarted the machine and noticed the same issue after 90 seconds. But if you read the log carefully, you will notice that the issue seems to be outputed twice. (looks the same). Probably two things are trying to write to that file (pm2-root.service).

Still waiting for the perfect solution (done by the person that really knows how to fix this in a proper manner), but for those that have this issue, any of these options seem to work just fine.

answered Aug 02 '20 at 00:11

Alex Laz

339
2
6

https://github.com/DataDog/datadog-agent/issues/5207 – Slawa Oct 07 '20 at 13:28
I think it's not safe to disable SE Linux – Epexa Nov 21 '20 at 05:39
3

Option 2 seems to have worked for me. Can anyone explain how this works? – naveed Nov 22 '20 at 19:08
1

Used option 2 then rebooted the server. Problem solved. Any security concerns? idk how this works either. I'm on Ubuntu though btw – mrg95 Jun 25 '21 at 06:07
I had to apply option 2 plus the following steps: Step 1: stop pm2 using systemctl / Step 2: sudo npm i -g pm2@latest / Step 3: sudo systemctl daemon-reload / Step 4: start pm2 again using systemctl – carlosrafaelgn Sep 30 '21 at 20:06
Thank you! Option 2 worked for me too on ubuntu. After commenting the line, i did "pm2 update" and then it was fine. – rya May 10 '22 at 15:02
Option 2 worked for me – Neophear May 25 '22 at 08:00

score 0 · Answer 5 · answered Dec 02 '20 at 08:50

0

I've had this problem (on Debian), when for some reason two "PM2 God Daemon" processes (not threads) were launched, so they conflicting with each other.

Killing one of them solved the issue.

answered Dec 02 '20 at 08:50

mortiy

669
7
9

PM2 keeps getting killed every 90 seconds on centos 8

5 Answers5