What is the ideal way to loop API requests with fetch?

Question

I'm relatively new to working with NodeJS, and I'm doing a practice project using the Youtube API to get some data on a user's videos. The Youtube API returns a list of videos with a page token, to successfully collect all of a user's videos, you would have to make several API requests, each with a different page token. When you reach the end of these requests, there will be no new page token present in the response, so you can move on. Doing it in a for, or while loop seemed like the way to handle this, but these are synchronous operations that do not appear to work in promises, so I had to look for an alternative

I looked at a few previous answers to similar questions, including the ones here and here. I got the general idea of the code in the answers, but I couldn't quite figure out how to get it working fully myself. The request I am making is already chained in a .then() of a previous API call - I would like to complete the recursive fetch calls with new page tokens, and then move onto another .then(). Right now, when I run my code, it moves onto the next .then() without the requests that use the tokens being complete. Is there any way to stop this from happening? I know async/await may be a solution, but I've decided to post here just to see if there are any possible solutions without having to go down that route in the hope I learn a bit about fetch/promises in general. Any other suggestions/advice about the way the code is structured is welcome too, as I'm pretty conscious that this is probably not the best way to handle making all of these API calls.

Code :

let body = req.body
let resData = {}
let channelId = body.channelId
let videoData = []
let pageToken = ''

const fetchWithToken = (nextPageToken) => { 
            
    let uploadedVideosUrlWithToken = `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${uploadedVideosPlaylistId}&pageToken=${nextPageToken}&maxResults=50&key=${apiKey}`

    fetch(uploadedVideosUrlWithToken)
    .then(res => res.json())
    .then(uploadedVideosTokenPart => {
        let {items} = uploadedVideosTokenPart
        videoData.push(...items.map(v => v.contentDetails.videoId))
        pageToken = (uploadedVideosTokenPart.nextPageToken) ? uploadedVideosTokenPart.nextPageToken : ''
            if (pageToken) {
                fetchWithToken(pageToken)
            } else {
                // tried to return a promise so I can chain .then() to it?
                // return new Promise((resolve) => {
                //     return(resolve(true))
                // })
            }
    })
}
const channelDataUrl = `https://youtube.googleapis.com/youtube/v3/channels?part=snippet%2CcontentDetails%2Cstatistics&id=${channelId}&key=${apiKey}`

// promise for channel data
// get channel data then store it in variable (resData) that will eventually be sent as a response, 
// contentDetails.relatedPlaylists.uploads is the playlist ID which will be used to get individual video data.

fetch(channelDataUrl)
    .then(res => res.json())
    .then(channelData => {
        let {snippet, contentDetails, statistics } = channelData.items[0]
        resData.snippet = snippet
        resData.statistics = statistics
        resData.uploadedVideos = contentDetails.relatedPlaylists.uploads
        return resData.uploadedVideos
    })
    .then(uploadedVideosPlaylistId => {

        // initial call to get first set of videos + first page token

        let uploadedVideosUrl = `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${uploadedVideosPlaylistId}&maxResults=50&key=${apiKey}`

        fetch(uploadedVideosUrl)
            .then(res => res.json())
            .then(uploadedVideosPart => {
                let {nextPageToken, items} = uploadedVideosPart
                videoData.push(...items.map(v => v.contentDetails.videoId))
                // idea is to do api calls until pageToken is non existent, and add the video id's to the existing array.
                fetchWithToken(nextPageToken)

            })

        })
        .then(() => {

            // can't seem to get here synchronously - code in this block will happen before all the fetchWithToken's are complete - need to figure this out

        })

Thanks to anyone who takes the time out to read this.

Edit:

After some trial and error, this seemed to work - it is a complete mess. The way I understand it is that this function now recursively creates promises that resolve to true only when there is no page token from the api response allowing me to return this function from a .then() and move on to a new .then() synchronously. I am still interested in better solutions, or just suggestions to make this code more readable as I don't think it's very good at all.

    const fetchWithToken = (playlistId, nextPageToken) => { 
            
    let uploadedVideosUrlWithToken = `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${playlistId}&pageToken=${nextPageToken}&maxResults=50&key=${apiKey}`


    return new Promise((resolve) => {
        resolve( new Promise((res) => { 
            fetch(uploadedVideosUrlWithToken)
                .then(res => res.json())
                .then(uploadedVideosTokenPart => {
                    let {items} = uploadedVideosTokenPart
                    videoData.push(...items.map(v => v.contentDetails.videoId))
                    pageToken = (uploadedVideosTokenPart.nextPageToken) ? uploadedVideosTokenPart.nextPageToken : ''
                        // tried to return a promise so I can chain .then() to it?
                        if (pageToken) {
                            res(fetchWithToken(playlistId, pageToken))
                        } else {
                            res(new Promise(r => r(true)))
                        }
                    })
                }))
            })
}

"When you reach the end of these requests" it's your line from question . Do you have any idea about how many requests going to come. or it's just you will only know after making some nth API call. — kumar kundan, Dec 22 '20 at 07:43
I could figure out how many requests are going to be needed - the initial API call gets the total amount of videos. I could divide this by max results and figure out how many calls I'd need to make. — lsg93, Dec 22 '20 at 14:07

omeanwell · Accepted Answer · 2020-12-22T17:34:00.437

You would be much better off using async/await which are basically a wrapper for promises. Promise chaining, which is what you are doing with the nested thens, can get messy and confusing...

I converted your code to use async/await so hopefully this will help you see how to solve your problem. Good luck!

Your initial code:

let { body } = req
let resData = {}
let { channelId } = body
let videoData = []
let pageToken = ''

const fetchWithToken = async (nextPageToken) => {
  const someData = (
    await fetch(
      `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${uploadedVideosPlaylistId}&pageToken=${nextPageToken}&maxResults=50&key=${apiKey}`,
    )
  ).json()

  let { items } = someData
  videoData.push(...items.map((v) => v.contentDetails.videoId))
  pageToken = someData.nextPageToken ? someData.nextPageToken : ''
  if (pageToken) {
    await fetchWithToken(pageToken)
  } else {
        // You would need to work out
  }
}

const MainMethod = async () => {
  const channelData = (
    await fetch(
      `https://youtube.googleapis.com/youtube/v3/channels?part=snippet%2CcontentDetails%2Cstatistics&id=${channelId}&key=${apiKey}`,
    )
  ).json()

  let { snippet, contentDetails, statistics } = channelData.items[0]
  resData.snippet = snippet
  resData.statistics = statistics
  resData.uploadedVideos = contentDetails.relatedPlaylists.uploads
  const uploadedVideosPlaylistId = resData.uploadedVideos

  const uploadedVideosPart = (
    await fetch(
      `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${uploadedVideosPlaylistId}&maxResults=50&key=${apiKey}`,
    )
  ).json()

  let { nextPageToken, items } = uploadedVideosPart
  videoData.push(...items.map((v) => v.contentDetails.videoId))
  await fetchWithToken(nextPageToken)
}

MainMethod()

Your Edit:

const fetchWithToken = (playlistId, nextPageToken) => {
  return new Promise((resolve) => {
    resolve(
      new Promise(async (res) => {
        const uploadedVideosTokenPart = (
          await fetch(
            `https://youtube.googleapis.com/youtube/v3/playlistItems?part=ContentDetails&playlistId=${playlistId}&pageToken=${nextPageToken}&maxResults=50&key=${apiKey}`,
          )
        ).json()

        let { items } = uploadedVideosTokenPart
        videoData.push(...items.map((v) => v.contentDetails.videoId))
        pageToken = uploadedVideosTokenPart.nextPageToken
          ? uploadedVideosTokenPart.nextPageToken
          : ''

        if (pageToken) {
          res(fetchWithToken(playlistId, pageToken))
        } else {
          res(new Promise((r) => r(true)))
        }
      }),
    )
  })
}

What is the ideal way to loop API requests with fetch?

1 Answers1