Pre-emptive: How to avoid deep callback hierarchy in Node? (Using Express)

Question

I just started experimenting with node (using Express to build a simple website with a MySql database).

I have basically ran with the application structure Express provides (which doesn't matter for the sake of this question). I have a file routes/index.js which exports the index function that is hit whenever a request is made for my home page. The contents of index.js are:

var db = require('../db');                                                      

exports.index = function(req, res){                                             
    db.getConnection(function(err, connection) {                                
        connection.query('SELECT * FROM test_table', function (err, rows) {     
            var templateVariables = {                                           
                title: 'Index Page',                                            
                response: rows[0].text                                          
            };                                                                  
            res.render('index', templateVariables);                             
        });                                                                     
        connection.end();                                                                                     
    });                                                                                                       
};

This is obviously a very preliminary and lightweight example, however on this particular GET request for the Index page, there is already a 3-deep set of callback functions. Each callbuck must live within the callback of the "parent", for it depends on the result (in a sequentially executed language/environment this would be obvious and trivial).

My question is that when building more complex and potentially very large applications, how does one avoid the issue of having massive nestings of callback functions? This will of course be the case when you have sequential dependency on logic. I know the philosophy of Node is to be asynchronous, but when it comes to waiting for data from the database(s) and say we're running 5 separate queries, what then? Would we just write a single multi-statement query as an atomic unit? Though this question isn't exclusive to databases.

score 2 · Answer 1 · answered Mar 24 '13 at 02:51

2

There's a nice general discussion on the issue here: http://callbackhell.com/

Also, many people use modules like Async to manage the flow control issues.

answered Mar 24 '13 at 02:51

Kevin Dente

25,430
7
43
47

score 2 · Answer 2 · answered Mar 24 '13 at 04:28

2

Since you mention by using Express you can use the next() as an alternative to callbacks.

app.get('/',function first(req,res,next){
   res.write('Hello ');
   res.locals.user = req.user;
   next();
   //Execute next handler in queue
});

app.get('/',function second(req,res,next){
   res.write('World!');
   //Use res.locals.user
   res.end();
});

//response shows Hello World!

The route handlers use extra parameter next and are executed in the order they are given, until one of them returns a response. next takes either no parameters at all or an error as a parameter. You can set the variable you want to pass into next function in the res.locals

answered Mar 24 '13 at 04:28

user568109

47,225
17
99
123

The use of `next()` would seem to make sense for having multiple handlers addressing the same (or same type of) request (as you pointed out). Though it doesn't seem like the facility the `next()` method provides would be a solution to the issue of arbitrarily tiered callbacks (As in the case of my database example). Would you agree? Thanks for the response though, its good to know. – DJSunny Mar 24 '13 at 04:32
Yes, it is not always possible to serialize complex callbacks. But this is helpful if you have to change the sequence of steps to handle the request. Think of it like response function broken into steps (easy to debug and you can reuse/modify step functions) – user568109 Mar 24 '13 at 04:45

score 1 · Answer 3 · answered Mar 24 '13 at 02:52

Use a Promise or Future library, such as Q (available on npm).

Quoting from Q's readme, promises let you turn this:

step1(function (value1) {
    step2(value1, function(value2) {
        step3(value2, function(value3) {
            step4(value3, function(value4) {
                // Do something with value4
            });
        });
    });
});

into this:

Q.fcall(step1)
.then(step2)
.then(step3)
.then(step4)
.then(function (value4) {
    // Do something with value4
}, function (error) {
    // Handle any error from step1 through step4
})
.done();

Every other solution I've seen to callback hell introduces trade-offs that simply seem a step backward. Asynchronous operations do not form the natural logical boundaries between larger operations, so if you factor functions or otherwise modularize along these boundaries you will get microfactored code.

One might think it would become easy to abuse/overuse such a library (Promise/Future). Are you of the opinion that it has its place, but one need carefully examine when to use it and when not to? — DJSunny, Mar 24 '13 at 03:00
@csjohn It's only abuse or overuse if there is a tradeoff. If you use it MORE you have even LESS callback hell to deal with, which is not overuse. But yes, chances are all of your code will look like this. However, it is easy to translate between Q (and I think any promise library) and callbacks, so you are not going to be stuck in this decision. — djechlin, Mar 24 '13 at 03:32

score 1 · Answer 4 · answered Mar 24 '13 at 03:47

One way that I am fond of doing is like this...

exports.index = function(req, res) {
  var connection

  db.getConnection(gotConnection)

  function gotConnection(err, _c) {
    connection = _c                                
    connection.query('SELECT * FROM test_table', gotData)
  }

  function gotData(err, rows) { 
    connection.end();        
    var templateVariables = {                                           
      title: 'Index Page', 
      response: rows[0].text     
    }
    res.render('index', templateVariables);
  }
});

You should also always handle errors as well in your code, I'm assuming you left them out to make the code easier to read here.

Pre-emptive: How to avoid deep callback hierarchy in Node? (Using Express)

4 Answers4