Node js Aggregate response of 10 async calls

Question

We have a requirement where we want to build api calls based on query param array for (contentId[]=1&contentId[]=2....and so on ), and then make async calls appending id to the api end point e.g http://xxxx/content/contentId.

Based on the response we need to aggregate and wrap the content with fields contentId, responsecode that we receive when we hit the individual api endpoints

{
 { "contentd": "1",
 "responsecode": 200,
"messsage": "",
content{
}
}

{ "contentd": "2",
 "responsecode": 200,
"messsage": "",
content{
}
...
}

We are using promise to do the same. I used promise all as below.

Promise.all(req.query.contentId
        .map(function (contentId) {
          console.log('processing the content id'+contentId);
          return getContent(contentId);
        }))
      .then(function (all_content) {

        //convert each resolved promised into JSON and convert it into an array.

        res.json(all_content.map(function (content) {

          return JSON.parse(content)
        }));
      })

      .catch(function(rej) {
      //here when you reject the promise
      console.log("404 received"+rej);


    }) ;
  } catch (e) {
    res.send(e.message);
  }
});

// this is the function that makes the call to the backend. Note usage of request promise
function getContent(contentId) {
  console.log('content id received to fetch the content'+contentId);
  var options = {
    uri: 'http://xxxxx/api/content/'+contentId,
    headers: {
        'Authorization': 'Basic YWRtaW46YWRtaW4='
    },
    qs: {

      _format: 'json'

    }
  };
  return rp(options);
}

Problems -The problem we are facing is that for the calls when we get http status code as 200 the result comes fine. However, for http status code 404(not found) the processing is not done. It seems to fail fast.

Also, how do I insert own fields for response, status while processing the content response as above in JSON.parse. thanks

Answer 1

If you read about Promise.all(), that's what it does. If any promise you pass it rejects, it immediately rejects the overall Promise.all() promise and you don't get the other results.

If you want all the other results, you have to either catch any rejections earlier so that Promise.all() never sees them or use a different way of tracking all the promises that is designed to get all respones, whether any reject or not.

You can catch the rejection earlier to "hide" it form Promise.all() by doing something like this and you also have to skip the JSON.parse() of any error objects that are passed through.

Promise.all(req.query.contentId
    .map(function (contentId) {
      console.log('processing the content id'+contentId);
      // catch errors here and make the resolved value be the error object
      // so Promise.all() processing continues on the other requests
      return getContent(contentId).catch(e => e);
    }))
  .then(function (all_content) {

    //convert each resolved promised into JSON and convert it into an array.

    res.json(all_content.map(function (content) {
      // make sure we don't try to parse things that are already objects
      // like error objects
      if (typeof content !== "string") {
          return content;
      } else {
          return JSON.parse(content);
      }
    }));
  }).then(function(results) {
    // remove any error objects
    // and probably log them to so they aren't just silently eaten
    return results.filter(item => !(item instanceof Error));
  })

Then, later down stream while using this array of results, you need to skip any error objects in the results.

Alternatively, you could make the results into some sort of no-op result that wouldn't affect your downstream processing or you can build a different array that had all error results filtered out of it.

And, if you want some prewritten code that just gets all the responses (including error results), that is often called Promise.settle(). There are several implementations of a Promise.settle() that you can use here:

ES6 Promise.all() error handle - Is .settle() needed?

---

Also, how do I insert own fields for response, status while processing the content response as above in JSON.parse.

You can change this:

return JSON.parse(content);

to something like this:

let obj = JSON.parse(content);
obj.someOtherProp = 'myvalue';
return obj;

Answer 2

 return getContent(contentId);

Here you return the promise to Promise.all. so if the promise fails, it will trigger the Promise.all catch and all routes fail. So we need to catch earlier, so right there:

return getContent(contentId).catch( e => e);

That means in case of an error, proceed with the error as result. As the JSON.parse would fail then, you could return a json string instead:

return getContent(contentId).catch( e => '{"error":500}');

---

For the second question:

You may want to assign additional properties to the returned result:

return Object.assign( JSON.parse(content), {
  what: "ever"
});