Correct approach to getting all pages from API that utilizes a Link header (using JavaScript/NodeJS)

Question

I'm using a NodeJS module called node-github, which is a wrapper around the Github API, to get some statistics about certain users, such as their followers:

var getFollowers = function(user, callback) {
  github.user.getFollowers(user, function(err, res) {
    console.log("getFollowers", res.length);
    callback(err, res);
  });
};

...

getFollwers({user: mike}, function(err, followers) {
  if(err) {
    console.log(err);
  }
  else {
    console.log(followers);
  }
});

Apparently, Github limits call results to a maximum of 100 (via the per_page parameter), and utilizes the Link header to let you know a 'next page' of results exists.

The module I'm using provides several easy methods to handle the Link header, so you won't need to parse it. Basically, you can call github.hasNextPage(res) or github.getNextPage(res) (where res is the response you received from the original github.user.getFollowers() call)

What I'm looking for is the right approach/paradigm to having my function return all results, comprised of all pages. I dabbled a bit with a recursive function, and though it works, I can't help but feeling there may be a better approach.

This answer could serve as a good approach to handling all future Link header calls - not just Github's - if the standard catches on.

Thanks!

Have you considered creating a custom readable stream which emits followers? — AndyD, Dec 11 '12 at 01:29

score 0 · Answer 1 · answered Dec 11 '12 at 21:59

Finally, resorted to recursion (remember the 2 great weaknesses of recursion: maintaining it, and explaining it :)). Here's my current code, if anyone is interested:

var getFollowers = function(callback) {
  var followers = []
  ,  getFollowers = function(error, result) {
      followers = followers.concat(result);
      if(github.hasNextPage(result)) {
        github.getNextPage(result, getFollowers);
      }
      else {
        callback(error, followers);
      }
    };              

  github.user.getFollowers(options, getFollowers);
};

But, I found out that if you just need the total number of followers, you can use the getLastPage function to get the number of followers on the last page and then

total_num_of_followers = num_of_followers_on_last_page + (total_num_of_pages * results_per_page)

Correct approach to getting all pages from API that utilizes a Link header (using JavaScript/NodeJS)

1 Answers1