How to ensure Node.js keeps running after MonogDB connection drops?

Question

I have an error-handling middlware in Express that tries to catch all incoming errors:

app.use(function(err, req, res, next){
  console.error(err.stack);
  res.status(500);
  res.render('500.jade');
});

But for some reason whenever I close mongod process, my application crashes with the following stack trace:

Error: failed to connect to [localhost:27017]
    at null.<anonymous> (/<hidden>/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:540:74)
    at EventEmitter.emit (events.js:106:17)
    at null.<anonymous> (/<hidden>/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:140:15)
    at EventEmitter.emit (events.js:98:17)
    at Socket.<anonymous> (/<hidden>/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection.js:478:10)
    at Socket.EventEmitter.emit (events.js:95:17)
    at net.js:441:14
    at process._tickCallback (node.js:415:13)

Process finished with exit code 8

I have tried using the following configuration, but it didn't help me:

var options = {
  server:{
    auto_reconnect: true,
      poolSize: 10,
      socketOptions:{
      keepAlive: 1
    }
  },
  db: {
    numberOfRetries: 10,
    retryMiliSeconds: 1000
  }
}

mongoose.connect(config.db, options);

One might wonder "why would you want your application to be running if it's essentially no longer functioning without a database connection?". I don't want it to crash and restart itself. Supervisor and Forever modules seem to stop trying to reconnect after a certain number of tries, thus leaving your application hanging in a crashed state.

Ideally, when MongoDB crashes I would like to display a 500.jade error page to users, meanwhile server should keep trying to reconnect to the database every 10 seconds. Once reconnected, resume all normal operations.

EDIT: None of the solutions posted below have worked for me, with the exception of domains.

score 6 · Accepted Answer · answered Dec 19 '13 at 19:13

6

Try initializing the DB in a domain, that would catch the errors without crashing

var d = require('domain').create();

d.on('error', function(er) {
    console.log('Oh no, something wrong with DB');
});

d.run(function() {
    mongoose.connect(config.db, options);
});

It's generally a good idea to let the server restart when something crashes, but as you're already aware of this and want to avoid it, wrapping whatever fails in domains is the way to do it.

answered Dec 19 '13 at 19:13

adeneo

312,895
29
395
388

1

Awesome, I have never used Node.js domains before. Learned something new today ;). Nice and elegant solution. – Sahat Yalkabov Dec 19 '13 at 19:21
1

@TwilightPonyInc. - domains where added not to long ago, and are still considered somewhat unstable, but I have been using something similar with Cassandra as DB in production for some time now, and have never had an issue with it, if the DB fails node keeps running, and when the DB is available again, it's like nothing ever happened, I've looked everywhere for leaks, and it seems fine to me. I haven't really tried it with Mongo, but it should be the same in principle. – adeneo Dec 19 '13 at 19:27
Just don't start wrapping internal node processes in domains, as that could actually lead to problems and serious leaks. – adeneo Dec 19 '13 at 19:28
I don't recommend domains with any Mongoose stuff yet - you have to wrap all your queries in it. Waiting for Mongoose support. – cyberwombat Dec 20 '13 at 07:13
@Yashua that's the only solution that seems to work. Middleware and `on.('close')` solutions don't work for me; mongoose still crashes the server as soon as I close `mongod`. It's almost as if it ignores my error handling code. – Sahat Yalkabov Dec 21 '13 at 18:34
@TwilightPonyInc. I would not use the on close event. I would watch the connection as illustrated in my answer – cyberwombat Dec 21 '13 at 18:41
1

There will be no special support for any database in domains, or anything else, as domains are just a way to capture any error, even fatal Node errors, without Node crashing. As such, domains should be used with caution as some errors can lead to unexpected behaviour if Node is not restarted. Generally, database errors are not fatal in Node, and in my experience won't cause issues if Node is not restarted, and as such they are a perfect example of why domains was added to begin with; to wrap code that might fail so as to not crash the entire Node proccess during failure. – adeneo Dec 21 '13 at 18:49

score 5 · Answer 2 · answered Dec 20 '13 at 07:11

This is what I do to deal with Mongo failing - add it as middleware. It will also try to reconnect again.

// Handler in case Mongo  goes down
app.use(function(req, res, next) {
  // We lost connection!
  if (1 !== mongoose.connection.readyState) {

    // Reconnect if we can
    mongoose.connect(config.db, options);
    res.status(503);
    throw new Error('Mongo not available');
  }
  next();

});

This assume you have a standard 50x error handler somewhere that will show a nice page to the user.

The reconnect is there for the next user/page load to check if it's back up. Works nice.

score 0 · Answer 3 · answered Dec 19 '13 at 19:23

0

The MongoDB driver issues a "close" event when the connection is lost. You can tap on this event and flag the DB as unavailable and write any logic based on that.

db.on('close', function() {
  // handle the disconnect
  dbAvailable = false;
});

I've got a small NPM module that handles this if you are interested: https://npmjs.org/package/mongoconnect

answered Dec 19 '13 at 19:23

Hector Correa

26,290
8
57
73

I can successfully catch this event by doing a simple `console.log()` but after about 1second, the server crashes with `throw er; // Unhandled 'error' event`. Am I doing something wrong here? (Using latest version of Express and Mongoose). Similar to: http://stackoverflow.com/questions/17809285/mongoose-times-out-and-throws-exception – Sahat Yalkabov Dec 19 '13 at 19:36

How to ensure Node.js keeps running after MonogDB connection drops?

3 Answers3

Linked