0

I'm using AWS EC2 instance where my node server is running

node - v10.16.3, npm - v6.11.3, mongo - v4.2.0,

Initially, I had these issues with my server where it logged me this error

{ MongoNetworkError: failed to connect to server [localhost:27017] on first connect [MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017]
    at Pool.<anonymous> (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/topologies/server.js:431:11)
    at Pool.emit (events.js:198:13)
    at Pool.EventEmitter.emit (domain.js:448:20)
    at connect (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:557:14)
    at makeConnection (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connect.js:39:11)
    at callback (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connect.js:261:5)
    at Socket.err (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connect.js:286:7)
    at Object.onceWrapper (events.js:286:20)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at emitErrorNT (internal/streams/destroy.js:91:8)
    at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'MongoNetworkError',
  errorLabels: [ 'TransientTransactionError' ],
  [Symbol(mongoErrorContextSymbol)]: {} }

I restarted my server after seeing this log and then my server worked perfectly fine with no issues for a few weeks. But then I saw other issues with Mongo which logged me that the Topology was Destroyed. 

{ MongoError: Topology was destroyed
    at initializeCursor (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/cursor.js:596:25)
    at nextFunction (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/cursor.js:456:12)
    at Cursor.next (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/cursor.js:766:3)
    at Cursor._next (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:216:36)
    at nextObject (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/operations/cursor_ops.js:192:10)
    at next (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/operations/cursor_ops.js:171:3)
    at executeOperation (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/utils.js:416:24)
    at Cursor.next (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/cursor.js:258:10)
    at findOne (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/operations/collection_ops.js:596:10)
    at executeOperation (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/utils.js:416:24)
    at Collection.<anonymous> (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/collection.js:989:12)
    at Collection.deprecated [as findOne] (/var/www/html/node_modules/mongoose/node_modules/mongodb/lib/utils.js:677:15)
    at NativeCollection.(anonymous function) [as findOne] (/var/www/html/node_modules/mongoose/lib/drivers/node-mongodb-native/collection.js:158:28)
    at NodeCollection.findOne (/var/www/html/node_modules/mquery/lib/collection/node.js:42:19)
    at model.Query.Query.findOne (/var/www/html/node_modules/mquery/lib/mquery.js:2025:20)
    at model.Query.<anonymous> (/var/www/html/node_modules/mongoose/lib/query.js:2058:22) name: 'MongoError', [Symbol(mongoErrorContextSymbol)]: {} }

I made some changes to the Mongo connection options setting like this

const options = { socketTimeoutMS: 30000, keepAlive: true, connectTimeoutMS: 30000, useNewUrlParser: true, useFindAndModify: false, useUnifiedTopology: true };

After changing the settings my server worked fine for some time and then then it goes unstable and unresponsive. I need to restart it manually every time to keep it alive

 (node:28639) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 31 finish listeners added. Use emitter.setMaxListeners() to increase limit

I have also had this memory leak issue in between these error logs.

{ MongoNetworkError: connection 1 to localhost:27017 timed out
    at Socket.<anonymous> (/var/www/html/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:335:7)
    at Object.onceWrapper (events.js:286:20)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at Socket._onTimeout (net.js:442:8)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
  name: 'MongoNetworkError',
  errorLabels: [ 'TransientTransactionError' ],
  [Symbol(mongoErrorContextSymbol)]: {} }

When my server gets unresponsive or unstable and if an API is called this is what I get in response

BadRequestError: request aborted
    at IncomingMessage.onAborted (/var/www/html/node_modules/raw-body/index.js:231:10)
    at IncomingMessage.emit (events.js:198:13)
    at IncomingMessage.EventEmitter.emit (domain.js:448:20)
    at abortIncoming (_http_server.js:448:9)
    at socketOnClose (_http_server.js:441:3)
    at Socket.emit (events.js:203:15)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at TCP._handle.close (net.js:606:12)

I'm not sure where's the issue exactly came from or what do I need to change to keep it alive. Please help me out with this issue since I cannot understand whether it is memory leak issue or server overloaded.

Karthik Ashokkumar
  • 59
  • 8
  • From couple of errors it meant to be your EC2 instance not able to reach mongoDB but it should not be the case, it should be consistent then - where you need to do ip whitelisting, Did you try to check when this has happened whether you're able to reach that mongoDB instance & seems like your mongoDB lies on same EC2 why does it have to be that way, why would a running instance go down, did you try to check mongoDB logs or EC2 processes ? If not if everything looks good then this could be an issue with your code where you're overloading things, what are you doing here ? – whoami - fakeFaceTrueSoul Dec 24 '19 at 05:40
  • Yes, I can reach my MongoDB server when I had this issue and the MongoDB server also lies in the same instance. There are other node servers running on the same instance which never had this issue till now expect this one. I also tried this solution https://stackoverflow.com/questions/41394850/mongoerror-connection-0-to-localhost27017-timed-out but it didn't work for me. – Karthik Ashokkumar Dec 24 '19 at 06:24
  • Karthik - There could be many reasons why this can happen, unless we see entire code/someone suggest from out of experience or a random guess can help you, but my best suggestion to you is to look at mongod logs at that particular time on what's happening & also look at node.js code on DB connectivity + Db transactions(any lag in query results or too much of data being processed or even maybe mongo primary instance getting crashed & it might take few seconds to get primary elected meanwhile your connections might be failing - best thing is to look at DB audit & also DB server logs) – whoami - fakeFaceTrueSoul Dec 24 '19 at 06:29
  • I would not suggest both servers to be in same instance, if you can move, move you mongoDB cluster to smoother in same VPC or entirely use mongoDB Atlas, check at your code for possible memory leaking scenarios on these particular calls.. – whoami - fakeFaceTrueSoul Dec 24 '19 at 06:31

0 Answers0