Create a pagination with Github API and express

Question

I'm currently creating one of my first small apps in Express and Node, which allows me to browse through all the repository from github, using their API. It all works fine for now, but of course some of the keywords i put into the search are giving me 17k or more result back, depending on how popular my search term is. The problem now is:

How to create a pagination? By default, the API returns 30 results PER PAGE, which can be changed to my liking, if i only want one result per page.The information, how many pages are available, can be found in the HTTP header, as far as i understand. I understand from their docs that i need to extract the link information from the http header like so:

curl -I "https://api.github.com/search/repositories?q=tetris"

results in:

HTTP/1.1 200 OK
Server: GitHub.com
Date: Fri, 19 Jan 2018 14:55:44 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 170353
...
X-GitHub-Media-Type: github.v3; format=json
Link: <https://api.github.com/search/repositories?q=pokemon&page=2>; rel="next", <https://api.github.com/search/repositories?q=pokemon&page=34>; rel="last"

Interesting for me is the link part, it gives me information about the next page, and how many there are in general. All the tuts i found right now are based results from a database and extracted info coming from tables or schemas. I just don't quite understand how I parse the curl information into express now, or what other possibilites I have. I tried with req.headers, which gave me

{ host: 'localhost:3000',
  connection: 'keep-alive',
  'upgrade-insecure-requests': '1',
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36',
  accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
  referer: 'http://localhost:3000/github',
  'accept-encoding': 'gzip, deflate, br',
  'accept-language': 'de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7' }

which is not quite what i was hoping for.

My current route look likes this:

app.get('/github/results', function (req, res) {
var query = req.query.gitHubsearch;
var options = {headers: {'User-Agent':'request'}};
var pageSize = 25;
var apiCall = 'https://api.github.com/search/repositories?q=' + query + '&order=desc+&per_page=' + pageSize;
request.get(apiCall, options, function (error, response, body) {
    if(!error && response.statusCode == 200) {
      var githubData = JSON.parse(body);
      console.log(req.headers);
      res.render("./github/results", {githubData: githubData});
    }
  });
});

I hope someone can give me a hint in what direction i have to look, or how i would create this kind of pagination now in NodeJS.

Answer 1

It looks like you're calling req.header which is using the request object of the request being made to your API, not the response from the github API call.

If you look at your app.get line you define req and res as per the express standard. This relates to the request being made to your server. The call to the request library then makes a HTTP call which then calls the callback function which populates error, response, body with details of your call up to github.

If you do console.log(response.headers) you should find the link item in there which you can access through response.headers.link.

It looks like you've just gotten a tiny bit confused between the request and response objects of your API vs the ones from the GitHub API is all. Here's some simple code which you should be able to run and see what you're expecting.

const request = require('request');

app.get('/github/result', function(req, res) {
    const endpoint = "https://api.github.com/search/repositories?q=" + query + "&order=desc+&per_page=" + pageSize;

    request.get(
      endpoint,
      { headers: { "User-Agent": "request" } },
      function(err, res, body) {
        console.log(err);
        console.log(res.headers);
      }
    );
});