We have ReactJS + NodeJS backend app running in Google GCP App Engine, but what we are experiencing is server down time for few min randomly. We have info Logs and error logs in code almost every api and every major functions and all catch blocks. I have also put global exception catch block
process.on('uncaughtException', (error) => {
Logger.error('--- Exception -----', error);
});
I also have error handler middleware my expressjs
app.use(function(error, req, res, next) {
if (error && error.stack && error.message) {
//cluster.worker.disconnect();
Logger.error('---- Global Exception Handler : error ---- ', error);
}
// Any request to this server will get here, and will send an HTTP
// response with the error message 'woops'
res.json({ status: 'FAIL', message: error.message });
});
but whenever server restarts by itself i don't see any exception in logs, it doesn't even appear in process.on('uncaughtException' block also.
One observation is GCP App engine is auto scaling when there is more traffic, but app is going down whenever there more traffic despite auto scaling is happening.
I want to know how to handle this, how to debug this to identify where exactly the issue is, is the issue is at code level if so why none of the exceptions are caught? or the issue is with GCP?
our production server also goes down many times randomly.
I tried using longjohn and setting NODE_DEBUG environment variable to net to get some hint as suggested in other SO Post
Update 1 : We Ran Jmeter Performance test to check whether load on server is the main reason, as Rafael Lemos rightly pointed in comment, in our case "The instance exceeds the maximum memory for its configured instance_class" may be the reason behind server going down, with lot of trail and error, when we made minimum no of instances = 2 we are not observing server going down.
However jmeter tests are all running fine but when we do jmeter test as well as manual testing at a time , we are observing Socket Hangup Exception, this crash happens immediately as soon as we start using front end app, where as jmeter tests access our backend app. Both backend and front end are running in same app engine.
Kindly find our observations below
Now question is Why Socket Hangup is coming, when I explored this, I found a solution in some other SO post, to handle socket hangup exception I have put below code in app.js but its not working
app.use((req, res, next) => {
if(res.socket.destroyed) {
console.log('----- socketIsDestroyed-- ', res.socket.destroyed);
res.end();
} else {
console.log('----- socketNotDestroyed-- ', res.socket.destroyed);
next();
}
})
And in Google App Engine Sys logs I see following error log multiple times
GET 503 /readiness_check failReason:"app lameducked"
Update2: my app.yaml file , is issue with resource limits ?
runtime: nodejs
env: flex
automatic_scaling:
min_num_instances: 2
max_num_instances: 5
resources:
cpu: .5
memory_gb: 0.9
disk_size_gb: 10
env_variables:
<Many Environment Variables here>
Update3: when we increased resource limit we are observing frequency of server going down is lesser, updated resource limit are as follows. Is it all because of memory limit issues ?
resources:
cpu: 2
memory_gb: 7.5
disk_size_gb: 10
Logs recorded while Socket hangup exception is as below
A 2021-02-25T05:17:16Z 1|dev_client | Error: socket hang up
A 2021-02-25T05:17:16Z 1|dev_client | at createHangUpError (_http_client.js:323:15)
A 2021-02-25T05:17:16Z 1|dev_client | at Socket.socketCloseListener (_http_client.js:364:25)
A 2021-02-25T05:17:16Z 1|dev_client | at Socket.emit (events.js:203:15)
A 2021-02-25T05:17:16Z 1|dev_client | at TCP._handle.close (net.js:606:12)
A 2021-02-25T05:17:16Z 1|dev_client | Error: socket hang up
A 2021-02-25T05:17:16Z 1|dev_client | at createHangUpError (_http_client.js:323:15)
A 2021-02-25T05:17:16Z 1|dev_client | at Socket.socketCloseListener (_http_client.js:364:25)
A 2021-02-25T05:17:16Z 1|dev_client | at Socket.emit (events.js:203:15)
A 2021-02-25T05:17:16Z 1|dev_client | at TCP._handle.close (net.js:606:12)
A 2021-02-25T05:17:16.691Z GET 499 0 B 6 s Firefox 85 /api/volunteercampaigns/6037246c4de6880024f01ff3/prominentVolunteers GET 499 0 B 6 s Firefox 85
A 2021-02-25T05:17:16.700Z GET 499 0 B 6 s Firefox 85 /api/volunteercampaigns/6037246c4de6880024f01ff3/prominentVolunteers GET 499 0 B 6 s Firefox 85
A 2021-02-25T05:17:17Z PM2 | App name:dev_client id:1 disconnected
A 2021-02-25T05:17:17Z PM2 | App [dev_client:1] exited with code [0] via signal [SIGINT]
A 2021-02-25T05:17:17Z PM2 | App [dev_client:1] starting in -cluster mode-
A 2021-02-25T05:17:17Z PM2 | App name:dev_client id:1 disconnected
A 2021-02-25T05:17:17Z PM2 | App [dev_client:1] exited with code [0] via signal [SIGINT]
A 2021-02-25T05:17:17Z PM2 | App [dev_client:1] starting in -cluster mode-
A 2021-02-25T05:17:18Z PM2 | App [dev_client:1] online
A 2021-02-25T05:17:18Z PM2 | App [dev_client:1] online
