Youtube Api, how to parse all vids with a nextPageToken?

Question

I'm using Node.JS, and the "request" library https://github.com/request/request I read the documentation, as much as I can request only 50 videos from the channel, in response from the api server, there is a nextPageToken key, but I do not understand how to make the chain of requests for this nextPageToken to go through the whole chain.

`https://www.googleapis.com/youtube/v3/search?key=${key}&channelId=${channelId}&part=snippet,id&order=date&maxResults=50`;

I sketched out this code

request({url:url}, function(err, response, body){
    let data = JSON.parse(body);
    for(let i = 0; i < Math.fround(data.pageInfo.totalResults / maxResults); i++){
        let newUrl = url + '&pageToken=' + data.nextPageToken;
        request({url: newUrl}, function(err, response, body){
            newUrl = url + '&pageToken=' + JSON.parse(body).nextPageToken;
            console.log(JSON.parse(body).nextPageToken);
        })
    }
})

At channel +450 video, did not think of the best solution and take the result of the first request, and divide by the maximum number of requested video, I get 9-10, then it turns out 10 passes per cycle, and in theory each request should update the variable newUrl And after, again access to the server for new data, and a new nextPageToken.

How to be?

You don't have to calculate how many requests you have to make. When you receive a response that has a `nextPageToken`, you set that as the value of `pageToken` for your next request. You then repeat this process (e.g. in a `while` loop), always with the most recent `nextPageToken`, until you get a response that doesn't have `nextPageToken` set. Then you are at the end of the search results. — paolo, Apr 23 '17 at 21:02
Hi, I was thinking about what I would do through the loop, and check if the last reply from the server contains the contents of `nextPageToken`, but the problem lies in the fact that I can not execute requests, ie, the query chain that will wait for the response, and after Answer will be executed with new parameters ... — CHBS, Apr 24 '17 at 17:54

score 0 · Answer 1 · answered Apr 25 '17 at 20:33

Typically when dealing with API's its not fair to use a for loop, you may want to step through your requests slower. My below example uses simple recursion but you may want to even add a timeout when the function calls itself.

let url = 'formatted url'
let videos = [];

function worker = (token, url) {
  
  //use the standard url if there is no token
  if (!token) {
    request({url:url}, function(err, response, body){
      let data = JSON.parse(body);
      //do something with data
      
      //if there is a next page token run worker again.
      if (data.nextPageToken) {
        let token = data.nexPageToken;
        
        return worker(token, url);
      } else {
        console.log('fin');
      }
      
   } else {
     //if there is a token use a new url
     let nurl = 'url + page token';
     
     request({url: url}, function(err, response, body) {
      
      let data = JSON.parse(body);
      //do something with data
      
      //if there is a token run the worker again
      
      if (data.nextPageToken) {
        let token = data.nextPageToken;
        return worker(token, url);
      } else {
        console.log('fin');
      }
      
     })
     
   }
}

worker(null, url);

Youtube Api, how to parse all vids with a nextPageToken?

1 Answers1

Linked