Programmatically Obtain All YouTube Videos Belonging to a User

Question

Utilizing the YouTube Data API we can make a query to to obtain the first 50 (maximum amount of results obtainable with a single query) videos belonging to a user with the following request: https://www.googleapis.com/youtube/v3/search?key={access_key}&channelId={users_id}&part=id&order=date&maxResults=50&type=video

If there are more than 50 videos, then the resulting JSON will have a nextPageToken field, to obtain the next 50 videos we can append &pageToken={nextPageToken} to the above request, producing the next page of 50 videos. This is repeatable until the nextPageToken field is no longer present.

Here is a simple JavaScript function I wrote using the fetch API to obtain a single page of videos, specified by the nextPageToken parameter (or lack thereof).

function getUploadedVideosPage(nextPageToken) {
    return new Promise((resolve, reject) => {
        let apiUrl = 'https://www.googleapis.com/youtube/v3/search?key={access_key}&channelId={users_id}&part=id&order=date&maxResults=50&type=video';
        if(nextPageToken)
            apiUrl += '&pageToken=' + nextPageToken;

        fetch(apiUrl)
        .then((response) => {
            response.json()
            .then((data) => {
                resolve(data);
            });
        });
    });
}

Now we need a wrapper function that will iteratively call getUploadedVideosPage for as long as we have a nextPageToken. Here is my 'working' albeit dangerous (more on this later) implementation.

function getAllUploadedVideos() {
    return new Promise((resolve, reject) => {
        let dataJoined = [];
        let chain = getUploadedVideosPage();

        for(let i = 0; i < 20000; ++i) {
            chain = chain
            .then((data) => {
                dataJoined.push(data);

                if(data.nextPageToken !== undefined)
                    return getUploadedVideosPage(data.nextPageToken);
                else
                    resolve(dataJoined);
            });
        }
    });
}

The 'dangerous' aspect is the condition of the for loop, theoretically it should be infinite for(;;) since we have no predefined way of knowing exactly how many iterations to make, and the only way to terminate the loop should be with the resolve statement. Yet when I implement it this way it truly is infinite and never terminates.

Hence why I hard coded 20,000 iterations, and it seems to work but I don't trust the reliability of this solution. I was hopping somebody here can shed some light on how to go about implementing this iterative Promise chain that has no predefined terminating condition.

changing `resolve(dataJoined);` to `return resolve(dataJoined);` did not fix it from being stuck in an infinite loop. — L. Key, Dec 22 '18 at 01:18
Why do you have a _for_ loop at all? Is it not enough to just let recursion do its thing? — Wyck, Dec 22 '18 at 01:26
I speculated recursion might be the way to go, unfortunately my recursive skills are 'rusty'. If you can demonstrate the recursive approach I would be much obliged. — L. Key, Dec 22 '18 at 01:32
Sorry if my previous comment was vague. What I was getting at is that I would expect that getUploadedVideosPage would return the results of the page concatenated with the remaining results from calling getUploadedVideosPage with the next page token. No loop. Just that getUploadedVideosPage calls itself in the thenable. — Wyck, Dec 22 '18 at 01:33
@L.Key Jason Byrne has the right idea: getNextVideo calls getNextVideo after then promise returns in his example. — Wyck, Dec 22 '18 at 01:37

charlietfl · Accepted Answer · 2018-12-22T01:49:00.367

You can do this all with one function that calls itself if applicable.

You are also using an explicit promise construction anti-pattern wrapping fetch() in new Promise since fetch() already returns a promise

function getVideos(nextPageToken, results = []) {

  let apiUrl = 'https://www.googleapis.com/youtube/v3/search?key={access_key}&channelId={users_id}&part=id&order=date&maxResults=50&type=video';
  if (nextPageToken) {
    apiUrl += '&pageToken=' + nextPageToken;
  }

  // return fetch() promise
  return fetch(apiUrl)
    .then(response => response.json())
    .then(data => {
      // merge new data into final results array
      results = results.concat(data);

      if (data.nextPageToken !== undefined) {
        // return another request promise
        return getVideos(data.nextPageToken, results);
      } else {
        // all done so return the final results
        return results
      }
    });
}

// usage
getVideos().then(results=>{/*do something with all results*/})
           .catch(err=>console.log('One of the requests failed'));

Thank you for the very clean solution. I definitely have some more reading to do on the mechanisms of promises. Cheers! — L. Key, Dec 22 '18 at 01:54
So the main thing to focus on here is the return of either a promise to keep the chain going or return data to go to the very final `then()` — charlietfl, Dec 22 '18 at 01:59

Jason Byrne · Answer 2 · 2018-12-22T01:40:11.477

Get rid of the for loop altogether. Something like this:

function getAllUploadedVideos() {
    return new Promise((resolve, reject) => {
        let dataJoined = [];

        function getNextPage(nextPageToken) {
            return new Promise((resolve, reject) => {
                getUploadedVideosPage(nextPageToken)
                    .then((data) => {
                        dataJoined.push(data);
                        if(data.nextPageToken !== undefined) {
                            resolve(data.nextPageToken);
                        }
                        else {
                            reject();
                        }
                    })
                    .catch((err) => {
                        // Just in case getUploadedVideosPage errors
                        reject(err);
                    });
            });
        }

        getNextPage()
            .then((nextPageToken) => {
                getNextPage(nextPageToken);
            })
            .catch(() => {
                // This will hit when there is no more pages to grab
                resolve(dataJoined);
            });

    });
}

It's more of a "getNextPage" as opposed to "getNextVideo", but basically the right idea. Maybe "getNextVideos" It's really the next page of 50 videos that it's getting on each request. — Wyck, Dec 22 '18 at 01:38
Yeah name if whatever is appropriate. But this let's it keep pulling new pages until it runs out, without the need of arbitrary for iteration. — Jason Byrne, Dec 22 '18 at 01:39

Programmatically Obtain All YouTube Videos Belonging to a User

2 Answers2