How to combine data from different callbacks in Node.js?

Question

I have multiple functions that are called back with data as it is received/parsed, and I need the data in one call back matched up to the data from another call back, which happen at different times.

For example, I have the following code to receive emails using pipedrive/inbox (github) client and andres9/mailparser (github) mailparser:

// List newest 10 messages
client.listMessages(-5, function(err, messages){
    messages.forEach(function(message){

        var mailparser = new MailParser();

        // setup an event listener when the parsing finishes
        mailparser.on("end", function(mail_object){
            console.log("---------------------------------------------------------<<<");
            console.log("Text body:", mail_object.text); // How are you today?
            console.log(">>>---------------------------------------------------------");
        });

        console.log(message.UID + ": Subject: < " + message.title + "> Flags: <" + message.flags + ">");

        client.createMessageStream(message.UID).pipe(mailparser);
            //send the current email to the parser

    });
});

The problem is that the console shows the message titles and flags all one after another, first, then shows all the parsed bodies one after another:

884: Subject: <[eBook] Executive's Guide to the Top 20 CSCs> Flags: <>
888: Subject: <Free Cloud-based Vulnerability Scanner> Flags: <\Seen>
896: Subject: <Unexpected sign-in attempt> Flags: <\Seen>
902: Subject: <[Watch] Backoff POS Malware: Key Indicators of Compromise> Flags: <$NotJunk>
918: Subject: <test subject 1> Flags: <$NotJunk>

[... Then here are all the bodies ...]
[... Message #884's body ...]
[... Message #888's body ...]
[... and so on ..]

When I want

884: Subject: <[eBook] Executive's Guide to the Top 20 CSCs> Flags: <>
[... Then message #884's body ...]

888: Subject: <Free Cloud-based Vulnerability Scanner> Flags: <\Seen>
[... Then message #888's body ...]

Eventually I will have this app send the full parsed messages (along with their parsed headers) to a database and also to the browser, and I need the flags and other metadata/headers paired with the parsed body when I do this, not logically separated by callbacks.

What is the best practice in Node.js for accomplishing this? What's the standard way to deal with separate callbacks for the same logical entities and are there different ways to deal with this?

score 2 · Accepted Answer · answered Nov 07 '14 at 23:30

2

Just do all of the work in the end listener:

client.listMessages(-5, function(err, messages){
    messages.forEach(function(message){

        var mailparser = new MailParser();

        // setup an event listener when the parsing finishes
        mailparser.on("end", function(mail_object){
            console.log(message.UID + ": Subject: < " + message.title + "> Flags: <" + message.flags + ">");
            console.log("---------------------------------------------------------<<<");
            console.log("Text body:", mail_object.text); // How are you today?
            console.log(">>>---------------------------------------------------------");
        });

        client.createMessageStream(message.UID).pipe(mailparser);
            //send the current email to the parser
    });
});

That one is easy because your calls are nested. Sometimes you have multiple asynchronous operations, that you want to act on once they are all finished. In that case, use a promise library like bluebird and call Promise.all().

answered Nov 07 '14 at 23:30

gilly3

87,962
25
144
176

Wow that is embarrassingly easy! I'm not used to callbacks so I'm still learning. I didn't realize that the scope carried into the function called by "end". – justwondering Nov 07 '14 at 23:36
This is a closure. Additional reading: http://stackoverflow.com/questions/111102/how-do-javascript-closures-work – gilly3 Nov 07 '14 at 23:56
What if the mailparser.on("end"...) statement had been declared before client.listMessages(...), how would you get message.title and message.flags? Assuming such a scenario came up elsewhere when coding. – justwondering Nov 08 '14 at 00:22
If they are available, get the values from `mail_object`. Otherwise, add another `end` listener as a closure. If the values aren't available in `mail_object`, a closure is the way to go. You could call the predefined listener function from within a closure. – gilly3 Nov 08 '14 at 01:15

score 1 · Answer 2 · answered Nov 07 '14 at 23:26

Simply move the console.log statement that displays the subject into the callback that displays the body ie

mailparser.on("end",     function(mail_object){
            console.log(message.title);
console.log("---------------------------------------------------------<<<");
        console.log("Text body:", mail_object.text); // How are you today?
        console.log(">>>---------------------------------------------------------");

score 0 · Answer 3 · answered Feb 24 '16 at 21:36

adding to gilly3's excellent answer you may need to display the data from newest to oldest. My example builds an object using messages.length as starting key and then decrement it down for each message until you get to "1" which will be the latest/newest message. Its then easy to loop the object from 1 to lastkey to output correctly.

  // list newest 2 messages 
  client.listMessages(-2, function(err, messages){
            var newob = {};

            var i = messages.length;

            messages.forEach(function(message){

                var mailparser = new MailParser();

                mailparser.on("end", function(mail_object){

                           console.log(message.UID+'  '+message.title);

                        /* because of loop direction i=1 will be latest/newest message
                           to get last key we can use lastkey = Object.keys(newob).length;
                           then loop from 1 to lastkey to output newest to last message.
                        */
                            newob[i] = {};
                            newob[i]['UID'] = message.UID;
                            newob[i]['from'] = mail_object.from;
                            newob[i]['subject'] = mail_object.subject;
                            newob[i]['title'] = message.title;
                            newob[i]['text'] = mail_object.text;

                            // count down 'i' so we know when at last message (i=0)
                            i--;

                            //now we're at end we can send object
                            if(i===0){
                              endResponse(newob);
                            }                                   

                });

                      client.createMessageStream(message.UID).pipe(mailparser);

            });
  });

How to combine data from different callbacks in Node.js?

3 Answers3