Integrating asynchronous mongo call within an inner forEach loop

Question

I got two loops, the outer loops over the users and the inner one loops over the venueID's of each user. Within the inner loop I want to look up the venue and attach it to an array defined in the outer look (userItem). However because forEach is synchronous and the mongo database look up is asynchronous the result always remains empty. I've tried to integrate this answer but to no avail. How to do this?

ret = [];
users.forEach(function(user) {
    var userItem = user.getSanitised('ADM');

    userItem.venues = [];

    var tmp = [];

    userItem.adminVenueIds.forEach(function(adminVenueId){
        tmp.push(function(callback) {

            Venue.findOne({_id:adminVenueId}, function(error, venue) {

                callback(null, venue.toObject());
            });
        });

    });

    async.parallel(userItem.venues, function(err, result) {
        /* this code will run after all calls finished the job or
         when any of the calls passes an error */
        if (err)
            return console.log(err);

        userItem.venues.push(result);
    });
    ret.push(userItem);
});

Tried the following as well but doesn't work also

users.forEach(function(user) {

    var userItem = [];
    async.series({

            setUserItem : function(callback)
            {
                userItem = user.getSanitised('ADM');
                callback(null, 'OK');
            },
            setUserVenues : function(callback)
            {
                userItem.venues = [];
                user.adminVenueIds.forEach(function(adminVenueId,index) {

                    Venue.findOne({_id:adminVenueId}, function(error, venue) {

                        userItem.venues.push(venue.toObject());

                        if((index+1) == user.adminVenueIds.length)
                            callback(null, 'OK');

                    });
                });
            }

        },
        function(error, results) {

            if(error)
                winston.error(error);

            ret.push(userItem);

        }

    );


});

Datsik · Answer 1 · 2015-06-04T02:58:22.103

1

You could simply put an if statement (in your case put the conditional as the array length) then when the loop is done you could then make it continue doing its thing by calling a function to continue (or put your code in there, but it will start to look messy)

var ret = [];
var test = [];


for (var i = 0; i < 20; i++) {
    for (var x = 0; x < 20; x++) {
         setTimeout(function() {
            test.push("Test"+x);
            if (x === 20) {
                finishIt();
            }
        }, 300)
    }
}


function finishIt() {
    console.log(test);
    ret.push(test);
}

edited Jun 04 '15 at 02:58

answered Jun 04 '15 at 02:11

Datsik

14,453
14
80
121

that doesn't help, the venue lookup call remains async and with that venues remains empty – bicycle Jun 04 '15 at 02:17
@bicycle Which part of your code are you trying to save for last? The `userItem.venues.push(result);`? – Datsik Jun 04 '15 at 02:19
@bicycle I hope my new answer solves your problem. It's just an example but you can mod it obviously to make it work for you – Datsik Jun 04 '15 at 02:58
that's so dirty, is there really no other solution? With any other language this would be a 5 minute job. – bicycle Jun 04 '15 at 03:06
Thanks but I ended up with the solution I posted. I upvoted you though because you were the one that gave me the idea to use setTimeout :) – bicycle Jun 04 '15 at 08:51
1

@bicycle Hey man, I didn't mean for you to use setTimeout, it was just the only way I could give an example of a loop with an async callback with a bit of a delay, the main point I was getting to was to use the `if` to see if it was done looping, but none the less glad you figured it out – Datsik Jun 04 '15 at 09:13

score 0 · Accepted Answer · answered Jun 04 '15 at 03:07

0

I think you might want to look into using Mongoose. It is a NodeJS application layer on top of MongoDB that provides a more SQL like experience.

http://mongoosejs.com

answered Jun 04 '15 at 03:07

Jeffrey A. Gochin

964
6
18

yeah they are telling me to use async which is what i'm doing http://stackoverflow.com/a/17296329/1160952 – bicycle Jun 04 '15 at 03:13
So Async is basically lets you implement Promises, a good technique, but not what your are really trying to accomplish IMO. You seem to trying to do an SQL style join correct? – Jeffrey A. Gochin Jun 04 '15 at 03:21
in sql terms yes but not in reality. I'm just for each venueID looking up the venue details and putting them in the userItem dictionary. Can't believe this is so difficult. – bicycle Jun 04 '15 at 07:33
look at the project feature of mongo. – Jeffrey A. Gochin Jun 04 '15 at 12:13
Sorry I meant populate. Here is a link for how mongoose would handle it. http://mongoosejs.com/docs/populate.html – Jeffrey A. Gochin Jun 04 '15 at 12:31
1

Thanks, that indeed looks promising. I'll check it out in a few days and see how it goes :) – bicycle Jun 11 '15 at 12:39

bicycle · Answer 3 · 2015-06-04T08:52:24.170

I ended up with the following solution. It's dirty but I guess that's just nodejs being nodejs.

users.forEach(function(user) {
    var userItem = user.getSanitised('ADM');
    userItem.venues = [];


    user.adminVenueIds.forEach(function(adminVenueId) {

        Venue.findOne({_id:adminVenueId}, function(error, venue) {

            userItem.venues.push(venue.toObject());

        });
    });

    (function(){

        if(userItem.venues.length == user.adminVenueIds.length) {

            ret.push(userItem);

        } else {
            setTimeout(arguments.callee, 30);
        }
    })();
});

Integrating asynchronous mongo call within an inner forEach loop

3 Answers3