6

When my node js server is running via pm2, it has a higher memory usage reading than the actual memory heap in the application when inspected in DevTools. More so, the value under memory in pm2 slowly increases over time, possibly indicating some kind of memory leak. This slow increase in memory usage also cannot be observed in DevTools.

Any explanation and/or solutions to these two (seemingly) strange occurrences?

This is my DevTools

enter image description here

This is pm2 list

enter image description here

here is my javascript code

var SSE = require('sse');
var https = require('https');
var fs = require('fs');
var url = require('url');
var mysql = require('mysql');
var schedule = require('node-schedule');

var options = {
    key: fs.readFileSync('pathto/ssl.key'),
    cert: fs.readFileSync('pathto/ssl.crt'),
    ca: fs.readFileSync('pathto/ssl.ca-bundle')
};


var pool = mysql.createPool({
    connectionLimit: 100,
    host: "host",
    user: "user",
    password: "pass",
    database: "db"
});

async function connectandrun() {
    try {

        var server = https.createServer(options, function(req, res) {
            var queryData = url.parse(req.url, true).query;
            res.writeHead(200, {
                'Content-Type': 'text/event-stream',
                'Access-Control-Allow-Origin': '*',
                'Cache-Control': 'no-cache',
                'Connection': 'keep-alive',
                'Access-Control-Allow-Methods': 'GET,PUT,POST,DELETE,OPTIONS',
                'Access-Control-Allow-Headers': 'Origin, X-Requested-With, Content-Type, Accept'
            });

            if (queryData.connectionid) {
                var connecitonid = queryData.connectionid;
            } else {
                var connecitonid = "";
            }

            var myconection = "myconnecction" + connecitonid;
            var uuserjson = {};
            uuserjson[myconection] = {
                Data: {
                    Element: null
                }
            };

            schedule.scheduleJob('*/3 * * * * *', function() {
                var runninginstance = main(uuserjson, queryData, myconection, res).catch(console.error);
                runninginstance = null;
            });

            res.on("close", function() {
                res.end();
                uuserjson[myconection] = null;
                myconection = null;
                connecitonid = null;
            });
        });

        server.listen(3000, '0.0.0.0', function() {
            var sse = new SSE(server);
            sse.on('connection', function(client) {
                client.send('hi there!');
            });
        });

    } finally {}
}

connectandrun().catch(console.error);

async function main(uuserjson, queryData, myconection, res) {


    pool.getConnection(function(err, con) {
        if (err) {
            console.log(err);
        } else {
            con.query("MYSQL QUERY",
                function(err, result, fields) {
                    if (err) throw err;
                    if (result.length != 0) {
                        uuserjson[myconection] = {
                            Data: {
                                Element: result[0]
                            }
                        };

                        if (result[0]) {
                            res.write("retry: 30000\n\n" + "event: blanks\ndata: " + result[0] + "\n\n");
                        }
                    }

                    con.release();
                });
        }
    });

}
user2993497
  • 525
  • 1
  • 7
  • 21
  • and what about the memory leak? my application is always around 9-11MB. But pm2's memory seems to be increasing with time? – user2993497 Dec 25 '19 at 01:51
  • That I'm unsure of. It's hard to diagnose a memory leak without any code... It could be some pm2 overhead? – Matt Oestreich Dec 25 '19 at 02:02
  • If you run your app without pm2, just regular ol `node index.js` or whatever, does the same thing happen? – Matt Oestreich Dec 25 '19 at 02:02
  • that's the thing. It doesn't. and the values I see in the DevTools image above are the same. I'm going to put my code in the question. – user2993497 Dec 25 '19 at 02:05
  • What version of pm2 are you using? – Matt Oestreich Dec 25 '19 at 02:22
  • I'm using version 4.2.1 – user2993497 Dec 25 '19 at 02:24
  • There are a couple of threads I have found on this.. [See this answer](https://stackoverflow.com/a/34090834/10431732) [See this bug](https://github.com/Unitech/pm2/issues/4134) Not sure if either of those will help.. but worth a shot. – Matt Oestreich Dec 25 '19 at 02:25
  • The bug you referenced is interesting I'll look into it and comment my results here I added my js code also – user2993497 Dec 25 '19 at 02:31
  • After Googling around on this it def seems like something is up with pm2... and has been for a long time. [See the last comment here](https://www.reddit.com/r/node/comments/2vbn8i/pm2_alternatives/cogoksa?utm_source=share&utm_medium=web2xI) ..It may be worth giving something else a shot.. It looks like [Phusion Passenger](https://www.phusionpassenger.com/) may be a good alternative? Quickstart guide can be found [here](https://www.phusionpassenger.com/library/walkthroughs/start/nodejs.html) – Matt Oestreich Dec 25 '19 at 02:42
  • wow. it seems you're right. I'll do some testing and I'll let you know, so you can post an answer – user2993497 Dec 25 '19 at 02:44
  • 1
    Sounds good - I need to do some research as well since I have been using pm2 (but have memory limits set so if RAM researches a certain threshold I auto restart pm2).. so something like this would have been hard for me to catch. Nice find!! – Matt Oestreich Dec 25 '19 at 02:46
  • Been running for a little while now with your suggestion and it's okay. Going to leave it for a few hours (as one of my tests only saw an increase after a few hours) and I'll return. Thanks for the suggestion. Will also do some digging up into pm2 and this alleged issue – user2993497 Dec 25 '19 at 03:22
  • Are you running on Phusion Passenger or did you start pm2 with that `gc` flag? – Matt Oestreich Dec 25 '19 at 03:23
  • 1
    Okay, I can confirm there seemed to be some kind of leak on pm2. I tried it on Passenger and the memory stayed stagnant. In your answer, can you talk a little about the discussions and rumours out there about pm2 leaking? and provide the solutions / alternatives you found. Thanks for pointing that out – user2993497 Dec 25 '19 at 17:14
  • 1
    Nice work - I have just finished my write-up on this. Glad we (mostly you) were able to figure this out. Hi-Five for teamwork! lol – Matt Oestreich Dec 25 '19 at 18:32

2 Answers2

17

After teaming up with OP on this, it has been confirmed there is some sort of memory leak in PM2.

Please see the below 'write-up' on our findings:


The Issue:

  • Slowly over time PM2 uses more and more RAM
    • This points to some sort of memory leak

Evidence & Rumors:

  • When running the application without PM2, just using node myserver.js, there is no evidence of RAM slowly increasing over time
    • RAM remains flat
    • The application that was used to test this theory is a small web app, in order to minimize the chance that the code contains a memory leak, and that the leak is in fact coming from PM2
  • There have been 'rumors' and evidence of memory leaks in PM2 for quite some time now - going back as far as 4 years (this answer was written in December 2019)
  • Someone describing why they stopped using PM2
    • See last comment
  • Thread where others are describing similar issues
    • That do not occur when PM2 is NOT used
  • Bug filed on PM2 GitHub with a confirmed memory leak
    • This bug was filed this year [2019] and at the time of this writing has comments as recent as October, where someone describes how this is still an issue

PM2 Alternatives You Should Consider:

  • Phusion Passenger seems like a strong candidate for replacing PM2
    • `Quick-start guide can be found here
    • Phusion Passenger looks like the closest comparison to PM2 and is where I would start
    • No, I do not have any affiliation with Phusion Passenger...
  • Forever
    • Does not seem like it is actively being maintained (in some of the issues filed on their GitHub people recommend using other apps)
  • Nodemon
    • Not really 'apples to apples' with PM2, but hey, variety is the spice of life
  • Naught
    • Does not appear to be actively maintained and is kind of 'small' in terms of stars
  • Native Node.js cluster

PM2 Workarounds/Band-Aids: (!not recommended to rely on these in Production!)

Matt Oestreich
  • 8,219
  • 3
  • 16
  • 41
0

I had the same error. The problem was that the NextJS build was not completed successfully. I did not notice this and tried to run pm2. Use command:

npm run build

And make sure the build completes successfully.