2

Summary

I'd like to collate paginated output into an array using JavaScript's Fetch API recursively. Having started out with promises, I thought an async/await function would be more suitable.

Attempt

Here's my approach:

global.fetch = require("node-fetch");

async function fetchRequest(url) {
  try {
    // Fetch request and parse as JSON
    const response = await fetch(url);
    let data = await response.json();

    // Extract the url of the response's "next" relational Link header
    let next_page = /<([^>]+)>; rel="next"/g.exec(response.headers.get("link"))[1];

    // If another page exists, merge it into the array
    // Else return the complete array of paginated output
    if (next_page) {
      data = data.concat(fetchRequest(next_page));
    } else {
      console.log(data);
      return data;
    }

  } catch (err) {
    return console.error(err);
  }
}

// Live demo endpoint to experiment with
fetchRequest("https://jsonplaceholder.cypress.io/posts?_page=9");

For this demo, it should result in 2 requests which yield a single array of 20 objects. Although the data is returned, I can't fathom how to collate it together into an array. Any guidance would be really appreciated. Thanks for your time.

Solution #1

Thanks to @ankit-gupta:

async function fetchRequest(url) {
  try {
    // Fetch request and parse as JSON
    const response = await fetch(url);
    let data = await response.json();

    // Extract the url of the response's "next" relational Link header
    let next_page;
    if (/<([^>]+)>; rel="next"/g.test(response.headers.get("link"))) {
      next_page = /<([^>]+)>; rel="next"/g.exec(response.headers.get("link"))[1];
    }

    // If another page exists, merge its output into the array recursively
    if (next_page) {
      data = data.concat(await fetchRequest(next_page));
    }
    return data;
  } catch (err) {
    return console.error(err);
  }
}

fetchRequest("https://jsonplaceholder.cypress.io/posts?_page=9").then(data =>
  console.log(data)
);

For each page, subsequent calls are made recursively and concatenated together into one array. Would it be possible to chain these calls in parallel, using Promises.all, similar to this answer?

On a side note, any ideas why StackOverflow Snippets fails on the second Fetch?

Rishav
  • 55
  • 1
  • 11

1 Answers1

3

You need to wrap next_page in a condition, otherwise it will lead to type error on the last call (Since /<([^>]+)>; rel="next"/g.exec(response.headers.get("link")) will be null)

Before concating data, you need the promise to get resolved.

Making some minor changes to your code can result in the correct output:

global.fetch = require("node-fetch");

async function fetchRequest(url) {
  try {
    // Fetch request and parse as JSON
    const response = await fetch(url);
    let data = await response.json();

    // Extract the url of the response's "next" relational Link header
    let next_page;
    if(/<([^>]+)>; rel="next"/g.exec(response.headers.get("link")))
        next_page = /<([^>]+)>; rel="next"/g.exec(response.headers.get("link"))[1];

    // If another page exists, merge it into the array
    // Else return the complete array of paginated output
    if (next_page) {
      let temp_data = await fetchRequest(next_page); 
      data = data.concat(temp_data);
    }

    return data;
  } catch (err) {
    return console.error(err);
  }
}

// Live, demo endpoint to experiment
fetchRequest("https://jsonplaceholder.cypress.io/posts?_page=9").then(data => {
    console.log(data);
});
Ankit Gupta
  • 757
  • 7
  • 13
  • Thanks for such a detailed response, I really appreciate it! Instead of repeating the `RegExp.exec` in the conditions and logic, would it be more efficient to use a `RegExp.test` in the condition instead? Is it advisable to bypass `temp_data` by moving the `fetchRequest()` call directly into `data.concat()`? What do you think of this script's structure; can it be improved? Would it be possible to chain calls in parallel [similar to this](https://stackoverflow.com/q/60590700/8377918)? – Rishav Jun 12 '20 at 10:57
  • For chaining calls in parallel, you need to remove dependency of getting the next link from response's header. Do you need to depend on every response's `next` relational link? What I mean to say is if you know the starting point is `https://jsonplaceholder.cypress.io/posts?_page=8` and ending point is `https://jsonplaceholder.cypress.io/posts?_page=10`, you can create the intermediate links yourself, right? – Ankit Gupta Jun 12 '20 at 13:46
  • 1
    Yes you can definitely move `fetchRequest()` call directly into `data.concat()`. I wrote it that way just to give a clear understanding of what's going on. – Ankit Gupta Jun 12 '20 at 13:51