0

I am having a case of in Nodejs where there is a for loop with callback functions. A loop is called on an array, and for each value an update opearation is called with query and that value is replaced by the result of the query.

The structure is something like this:

console.log(body);

for (var i = 0; i < body.length; i++) {
    console.log(body[i]);
    QueryManager.query(lookup_query, [body[i]], function (err, query_result) {
        if (err) {
            console.log('Query failed', err);
            return;
        }

        query_result = JSON.parse(query_result);

        if(query_result.matrics.resultCount == 1)
            body[i] = query_result.results[0].id;

        console.log("Array value replaced for position:" + i);
    });
}

console.log("Array updated!");

The for loop can be completed with the async execution and then it should wait for all the callbacks to be completed. So the next statements should be in sync after the array is updated fully.

So console outputs for tracking loops should be like in this order:

["1","2","3","4","5"]
Array value replaced for position:0
Array value replaced for position:1
Array value replaced for position:3
Array value replaced for position:2
Array value replaced for position:4
Array updated!

Except the first and the last output, in between outputs can be in any order.

How can I achieve this?

EDIT: The whole thing is inside the http server request. So the array I am getting is from the user's POST. Mapping each value to the actual value from based on the key provided in the array. After all the values replaced in the array from its key to value, I need to prepare some external data based on that values. And then sending that data back to user in the response of that request.

So this should not block the request thread until the whole request is finished and response is send. It should be ready to listen for the new request.

kirtan403
  • 7,293
  • 6
  • 54
  • 97
  • I am not quite sure what you want to achieve? You just want to replace a position in your array and call code after your loop is done? – Clemens Himmer Feb 16 '16 at 11:56
  • As Clemens Himmer said we need to know what your end goal is and perhaps a little reasoning why you need to accomplish it. If all you really need to do is run code after all of your queries have finished then this could be done with collecting the promises from the queries into a collection and then using Promise.all(promiseCollection).then(). If you need Query 2 to wait on Q1 and Q3 to wait on Q2 then you will have to chain the queries like so, Q1(callBack).then(Q2...) I.E. some very ugly code. – Michael Hobbs Feb 16 '16 at 12:06
  • A perfect use case for **async.each or async.eachSeries** – Raf Feb 16 '16 at 12:14
  • @MichaelHobbs I do not need Q1(callback).then(Q2..) as all the queries are a saperated thing and not related to each other. Can you explain what Promise.all(..) do ?? – kirtan403 Feb 16 '16 at 12:31
  • Basically I am doing some kind of mapping. There comes an array in http request. I am processing mapping the values to the actual values from the database and than preparing some data based on the new array and then sending the response back to user. – kirtan403 Feb 16 '16 at 12:33

2 Answers2

1

Use async.forEachLimit with limit=1.

Example :

var tempArray = [1, 2, 3, 4, 5];

async.forEachLimit(tempArray, 1, function(no, callback) {
    console.log("Current number : " + no);
    callback();
}, function(error){
    callback(error);
});

This code will print as follow:

Current number : 1

Current number : 2

Current number : 3

Current number : 4

Current number : 5

  • Can you explain a bit what it does? or with an example? – kirtan403 Feb 16 '16 at 12:28
  • @kirtan403 have a look how the async.each has been used in this question http://stackoverflow.com/questions/35394416/nodejs-how-to-use-async-js-to-process-a-list-of-items-in-database/35395478#35395478 – Raf Feb 16 '16 at 12:36
  • Thanks ! I got the point. I might need this somewhere in my project ! :) – kirtan403 Feb 16 '16 at 12:51
0

Basically Promise.all waits for all promises in a collection to resolve and then will it's self resolve calling the .then() :

console.log(body);

var promiseCollection = [];
for (var i = 0; i < body.length; i++) {
    console.log(body[i]);
    promiseCollection.push(new Promise(function (resolve, reject) {
      QueryManager.query(lookup_query, [body[i]], function (err, query_result) {
          if (err) {
              console.log('Query failed', err);
              reject(weFailedAndCanReturnWhy);
              return;
          }

          query_result = JSON.parse(query_result);

          if(query_result.matrics.resultCount == 1)
              body[i] = query_result.results[0].id;

          console.log("Array value replaced for position:" + i);
          resolve(canReturnStuffHere);
      })})
    );
}

Promise.all(promiseCollection)
  .then(function(){
    console.log("Array updated!");
  });
kirtan403
  • 7,293
  • 6
  • 54
  • 97
Michael Hobbs
  • 1,663
  • 1
  • 15
  • 26
  • Thanks.. I understood..It will not block my server thread, right? And which lib is this? I am new to nodejs.. – kirtan403 Feb 16 '16 at 12:46
  • As to the lib I'm not entirely sure, it's part of a yo generator I use. I would assume it should just be part of Node as it is part of es6. And no it will not block as it is async. – Michael Hobbs Feb 16 '16 at 12:49
  • I checked, bluebird is the one that is used in our projects. – Michael Hobbs Feb 16 '16 at 13:05
  • Yup ! Me too finalized bluebird based on this discussion here : https://github.com/petkaantonov/bluebird/issues/381 – kirtan403 Feb 16 '16 at 13:07
  • Here, the callback function of query manager is called after the array updated function. Something is suspicious – kirtan403 Feb 16 '16 at 13:39
  • I don't understand, explain please. – Michael Hobbs Feb 16 '16 at 13:47
  • The output looks something like this : ["1","2","3","4","5"] Array updated! Array value replaced for position:0 ..... so on.. I think Promise executes the line but with each execution of that line there is a callback function which is executed after the query is completed. In between Array updated is called as it executed all the lines. – kirtan403 Feb 16 '16 at 13:51
  • I'm guessing that QueryManager does not return a promise. You will have to wrap it in a promise. Give me a few and I'll update the answer. – Michael Hobbs Feb 16 '16 at 13:53
  • Yes, sure.. One point -> that QueryManager is external lib's function. – kirtan403 Feb 16 '16 at 13:59
  • You should look at async.each or async.eachSeries, but I have not learned them yet. On that long list of things todo. – Michael Hobbs Feb 16 '16 at 14:03
  • Awesome! this tends to work now. Me too will add that to the list now ! – kirtan403 Feb 16 '16 at 14:04
  • You wanna add a `.catch(handler)` to your code in case any of the async calls fails the entire promise will fail. If you don't care about all calls to be successful then `Promise.some()` is the way to go. – Aukhan Feb 16 '16 at 14:11
  • @MichaelHobbs Promise.all is parallel, op wants serial exec – simon-p-r Feb 16 '16 at 15:28
  • @simon-p-r It seems like he wants a set of async task to complete followed by another async task that depends of the set of tasks. – Michael Hobbs Feb 16 '16 at 15:59
  • Nope he wanted the callbacks to run in order – simon-p-r Feb 16 '16 at 21:04
  • @simon-p-r He did not care: MichaelHobbs I do not need Q1(callback).then(Q2..) as all the queries are a saperated thing and not related to each other. Can you explain what Promise.all(..) do ?? – kirtan403 – Michael Hobbs Feb 16 '16 at 21:05
  • Promise all runs all async ops in no specific order, question says it must be in sync?!?! – simon-p-r Feb 16 '16 at 21:08