Iterate through Array, update/create Objects asynchronously, when everything is done call callback

Question

I have a problem, but I have no idea how would one go around this.

I'm using loopback, but I think I would've face the same problem in mongodb sooner or later. Let me explain what am I doing:

I fetch entries from another REST services, then I prepare entries for my API response (entries are not ready yet, because they don't have id from my database)
Before I send response I want to check if entry exist in database, if it doesn't:
- Create it, if it does (determined by source_id):
- Use it & update it to newer version
Send response with entries (entries now have database ids assigned to them)

This seems okay, and easy to implement but it's not as far as my knowledge goes. I will try to explain further in code:

//This will not work since there are many async call, and fixedResults will be empty at the end
var fixedResults = [];
//results is array of entries
results.forEach(function(item) {
    Entry.findOne({where: {source_id: item.source_id}}, functioN(err, res) {
        //Did we find it in database?
        if(res === null) { 
            //Create object, another async call here
            fixedResults.push(newObj);
        } else {
            //Update object, another async call here
            fixedResults.push(updatedObj);
        }
    });
});
callback(null, fixedResults);

Note: I left some of the code out, but I think its pretty self explanatory if you read through it.

So I want to iterate through all objects, create or update them in database, then when all are updated/created, use them. How would I do this?

score 1 · Accepted Answer · edited May 23 '17 at 12:29

1

You can use promises. They are callbacks that will be invoked after some other condition has completed. Here's an example of chaining together promises https://coderwall.com/p/ijy61g.

The q library is a good one - https://github.com/kriskowal/q

This question how to use q.js promises to work with multiple asynchronous operations gives a nice code example of how you might build these up.

edited May 23 '17 at 12:29

Community

1
1

answered Jul 16 '14 at 19:12

Jeff Storey

56,312
72
233
406

score 1 · Answer 2 · answered Jul 16 '14 at 19:17

1

This pattern is generically called an 'async map'

answered Jul 16 '14 at 19:17

aredridel

1,532
14
19

This would get a lot easier with promises, since you can return earlier and keep your code from being so deep. – aredridel Jul 16 '14 at 19:18
thanks for the comment, I checked both answers, yours and from Jeff Storey, I get both concepts. Is his solution better? – ewooycom Jul 16 '14 at 19:24
Promises are nice, but at the moment, cost a bit in terms of runtime overhead -- especially Q. – aredridel Jul 24 '14 at 17:09
There's also the cost of adding a dependency. It really depends on the context. – aredridel Jul 24 '14 at 17:10

score 0 · Answer 3 · answered Jul 16 '14 at 19:18

You could use async.map for this. For each element in the array, run the array iterator function doing what you want to do to each element, then run the callback with the result (instead of fixedResults.push), triggering the map callback when all are done. Each iteration ad database call would then be run in parallel.

score 0 · Answer 4 · answered Jul 16 '14 at 19:23

Mongo has a function called upsert.

http://docs.mongodb.org/manual/reference/method/db.collection.update/

It does exactly what you ask for without needing the checks. You can fire all three requests asnc and just validate the result comes back as true. No need for additional processing.

Iterate through Array, update/create Objects asynchronously, when everything is done call callback

4 Answers4