use fetch to load paginated data recursively

Question

I would like to fetch data from this API: "https://swapi.co/api/planets". The data is paginated like https://swapi.co/api/planets/?page=1, https://swapi.co/api/planets/?page=2....

I want to build an API which accepts url, page, and callback as arguments. It will fetch all of the data page by page, until it reaches the page specified in the arguments.

E.g.

function loadData(https://swapi.co/api/planets, 5, cb)

this will load data from page1, page2, page3, page4 and page5.

Here is my attempt however it doesn't work. Can someone please point me in the right direction ?

function fn(url, page, pages, cb) {
  return new Promise((resolve, reject) => {
    const endpoint = `${url}/?page=${page}`;
    fetch(endpoint).then(response => {
      if (response.status !== 200) {
        throw `${response.status} ${response.statusText}`;
      }
      response.json().then(data => {
        cb(data);
        if (page <= pages) {
          fn(url, page + 1, pages, cb)
            .then(resolve)
            .catch(reject);
        } else {
          resolve();
        }
      });
    });
  });
}

function cb(data) {
  console.log(data)
}

fn('https://swapi.co/api/planets', 1, 3, cb).then(() => {});

Follow up question: if instead of passing in a callback as the argument, I want to pass in an array to collect the data loaded, how should I tweak this API?

[What is the explicit promise construction antipattern and how do I avoid it?](https://stackoverflow.com/questions/23803743/what-is-the-explicit-promise-construction-antipattern-and-how-do-i-avoid-it) — Andreas, Mar 08 '20 at 18:41

score 2 · Accepted Answer · answered Mar 08 '20 at 18:50

As has been said before, you probably don't need to wait for each call to finish before doing the next, but if you really want to I just have to say I find it much easier to parse (as a human) using async / await.

I've done a quick basic version here in this snippet for you to see, with no error handling or checking the data is right. Maybe this approach might suit you better?

async function getPage(page) {
  let response = await fetch(`https://swapi.co/api/planets/?page=${page}`);
  let data = await response.json()
  return data;
}


async function getPages(startPage, endPage) {
  let currentPage = startPage;
  let finalArr = [];
  while(currentPage <= endPage) {
    const pageData = await getPage(currentPage);
    console.log('adding page', currentPage);
    finalArr.push(...pageData.results);
    currentPage++;
  }

  console.log(finalArr)
}

getPages(1, 3);

Hi thank you for the answer! Really learned a lot. I found this question is kind of similar to what I'm doing here. https://stackoverflow.com/questions/49129245/javascript-using-fetch-and-pagination-recursive but they are recursively creating a promise to solve it. Was it because every page are dependent on the previous fetched page so we have to do it that way? — Joji, Mar 08 '20 at 22:20

score 0 · Answer 2 · answered Mar 08 '20 at 18:46

Since each request does not depend on the previous, I would use Promise.all method instead of requesting each page when the previous request has completed.

You can do something like:

function loadData(url, pages, cb) {
    const urls = [];

    for (let i = 1; i <= pages; i++) {
        urls.push(`${url}/?page=${i}`);
    }

    return Promise.all(urls.map(url => fetch(url).then(response => response.json())))
        .then(responses => {
            var results = responses.reduce((accum, response) => [...accum, ...response.results], []);
            cb(results);
            return results;
        });
}

use fetch to load paginated data recursively

2 Answers2

Linked