How to capture results from end of FOR loop with Nested/Dependent APIs calls in Node JS

Question

This is my first JavaScript & Node project and I am stuck….

I am trying to call a REST API that returns a set of Post IDs... and based on the set of retrieved IDs I am trying to call another API that returns details for each ID from the first API. The code uses Facebook API provided by Facebook-NodeSDK.

The problem I am having is that the second API fires of in a FOR Loop…. As I understand the for loop executes each request asynchronously…. I can see both the queries executing however I can’t figure out how to capture the end of the second for loop to return the final result to the user…

Following is the code…

exports.getFeeds = function(req, res) {

    var posts = [];
    FB.setAccessToken(’SOME TOKEN');
    var resultLength = 0;


     FB.api(
        //ARG #1 FQL Statement
        'fql', { q: 'SELECT post_id FROM stream WHERE filter_key = "others"' },
        //ARG #2 passing argument as a anonymous function with parameter result
        function (result)
           {

               if(!result || result.error) {
                    console.log(!result ? 'error occurred' : result.error);
                    return;
                } //closing if handling error in this block

                    var feedObj
                    console.log(result.data);
                    console.log(result.data.length);

                        for (var i = 0; i<resultLengthj ; i++) {

                        (function(i) {
                            feedObj             = {};
                            FB.api( result.data[ i].post_id,  { fields: ['name', 'description', 'full_picture' ] },
    //                          fbPost is data returned by query
                                function (fbPost) {
                                    if(!fbPost || fbPost.error) {
                                        console.log(!fbPost ? 'error occurred' : result.error);

                                        return;
                                    }
    //                                else
                                        feedObj=fbPost;
                                        posts.push(feedObj);
                            });
                       })(i);
                    }// end for

           }//CLOSE ARG#2 Function

    );// close FB.api Function

NOTE I need to call…... res.Send(post)…. and have tried to call it at several places but just can’t get all the posts… I have removed the console statements from the above code…which have shown that the data is being retrieved...

Thanks a lot for your help and attention....

first, thanks for marking my answer accepted. Second, kudos and wise choice to learn the fundamentals before grabbing modules (I think too many reflexively grab modules and then lack the appreciation for what those modules are actually doing). Third, if you are going to look at libs at some point, I will put in a small plug for [Q promises](https://www.npmjs.org/package/q), with which I have no affiliation other than as a user of it. I have personally found it to be a great way to more easily deal with asynch operations. It is not as popular as Async, but not much less (#11 on NPM) — barry-johnson, Mar 06 '14 at 00:50
Thanks Barry... I think it would be helpful if you could share how you would implement the above problem using Q Promises...Only if you can make the time... — Vishal, Mar 06 '14 at 12:47
Vishal - I will do that when I have a few moments free. Question: is the FB library one you (or your team, etc) built or an existing one available on NPM. I ask because it seems to employ something other the 'standard' node callback with a `callback(err,data)` signature and just has the `callback(data)` signature. — barry-johnson, Mar 06 '14 at 15:15
In the interim, I have provided some examples in some other questions - you might find [this](http://stackoverflow.com/questions/22109487/nodejs-mysql-dump/22110015#22110015) and [this](http://stackoverflow.com/questions/22138759/how-to-convert-step-code-to-async-js-step-waterfall-this-parallel/22154290#22154290) helpful as overviews. — barry-johnson, Mar 06 '14 at 15:15
I use facebook-node-sdk... available on NPM... I think it uses the non standard convention because the module relies on FB to provide errors in the results... the idea is for then module to not get wrapped up in error code management and let the user parse it from the result set... reducing the module upgrades to a minimum — Vishal, Mar 06 '14 at 20:55

score 0 · Accepted Answer · answered Mar 02 '14 at 00:23

If you just stick res.send almost anywhere in your code, it will be certain to get called before your posts have returned.

What you want to do in a case like this is to declare a counter variable outside your for loop and set it to zero. Then increment it inside the for loop. Then inside your inner callback (in your case, the one that is getting called once for each post), you would decrement the counter and test for when it hits zero. Below I apply the technique to an edited version of your code (I didn't see what feedObj was actually doing, nor did I understand why you were using the immediately-invoked function, so eliminated both - please let me know if i missed something there).

var posts = [];
FB.setAccessToken(’SOME TOKEN');
var resultLength = 0;


FB.api(
    //ARG #1 FQL Statement
    'fql', { q: 'SELECT post_id FROM stream WHERE filter_key = "others"' },
    //ARG #2 passing argument as a anonymous function with parameter result
    function (result)
    {

        if(!result || result.error) {
            return;
        } //closing if handling error in this block

        var counter = 0;
        for (var i = 0; i<resultLengthj ; i++) {

            counter++;
            FB.api( result.data[ i].post_id,  { fields: ['name', 'description', 'full_picture' ] },
                function (fbPost) { // fbPost is data returned by query
                    if(!fbPost || fbPost.error) {
                        return;
                    }
                    posts.push(fbPost);
                    counter--;
                    if (counter === 0){
                        // Let's render that page/send that result, etc
                    }
                });

        }// end for

    }//CLOSE ARG#2 Function

);// close FB.api Function

Hope this helps.

Thanks barry I am going to try this... to answer your question... I used the immediately-invoked function because each call in the function gets executed as a async process and the value of i is inconsistent, as the timing varies... pls look at eg... http://stackoverflow.com/questions/18864964/how-to-handle-for-loop-in-node-js... yes feedObj is useless... residual of some debugging — Vishal, Mar 02 '14 at 05:56
tried your idea, worked perfectly... added the immediately-invoked function to get a good value for i so I could index the object with a number. — Vishal, Mar 02 '14 at 06:23
I understand what you are saying about the immediately invoked function, but the reason I factored it out is that in this case, `i` itself isn't referenced in the callback, it's only referenced in the invocation `result.data[i].post_id`. Of course, your code here might be an abbreviated version, in which case it makes perfect sense. Most of all, glad it helped in any case! — barry-johnson, Mar 02 '14 at 06:50

score 0 · Answer 2 · answered Mar 02 '14 at 02:35

Essentially you are wanting to do an async map operation for each id.

There is a really handy library for doing async operations on collections called async that has a async.map method.

var async = require('async'); 

FB.api(
    'fql', { q: 'SELECT post_id FROM stream WHERE filter_key = "others"' },
    //ARG #2 passing argument as a anonymous function with parameter result
    function (result) {

        async.map(
            result,

            function (item, callback) {
                FB.api(
                    item.post_id, 
                    { fields: ['name', 'description', 'full_picture' ] }, 
                    callback
                );
            },

            function (err, allPosts) {
                if (err) return console.log(err);

                // Array of all posts
                console.log(allPosts);
            }
        );

    }
);

You definitely don't need to use this, but it simplifies your code a bit. Just run npm install --save async in your project directory and you should be good to go.

thanks Derek... I looked at Async... I want to understand the constructs and principals of node before I went the utility module route...Having said that I will try the Async solution too... — Vishal, Mar 02 '14 at 05:59

How to capture results from end of FOR loop with Nested/Dependent APIs calls in Node JS

2 Answers2