Best way to make a http request and then potentially another one

Question

Title isn't so clear but to elaborate, I need to make a HTTP request to an API endpoint, and so far I'm using a function that looks something like this:

function getPostsFromAPI(argOne, argTwo) {
    const apiUrl = `https://www.exampleapi.com/v1/userposts`
    apiGet(`${apiUrl}?name=argOne&something=argTwo`).then(userPosts => {
        // do stuff with userPosts
        return userPostData
    }).catch(handleError)
}

However, the API response can include the following:

{
    //...
    "has_more": true,
    "next_offset": 10
}

In which case, I'd need to send the API call a second time, this time with the &offset=10 argument.

The promise would need to continue making API calls until has_more: true is no longer present. My initial thought would be to just re-run getPostsFromAPI() based on an if statement from inside itself, but I can't figure out how to make that work cleanly inside a promise. Ultimately, the promise should keep making requests until the API says that it's ran out of data to give (I'll implement my own limit).

What would be the best way to achieve this?

Can you use async/await? Doing so may simplify potential solutions. — user229044, May 09 '21 at 02:35
@meagar yep, it's a personal project using node so I'm free to use pretty much anything — kougami, May 09 '21 at 02:37
This might be helpful https://stackoverflow.com/a/38543075/294949 — danh, May 09 '21 at 02:45

user229044 · Accepted Answer · 2021-05-09T02:54:05.683

The algorithm to achieve this is much more obvious if you use async/await. You can just create an empty array, and gradually append to it in a loop until the server indicates there are no more results.

async function getPostsFromAPI(argOne, argTwo) {
    const apiUrl = `https://www.exampleapi.com/v1/userposts`

    let results = [];
    let offset = 0;

    while (true) {
        let response = await apiGet(`${apiUrl}?name=argOne&something=argTwo&offset=${offset}`);
        results = results.concat(response.records);
        if (response.has_more) {
            offset = response.next_offset;
        } else {
            return results;
        }
    }
}

If you can't use async/await and have to stick to promises, you can use recursion to have a method invoke itself each time a response indicates there are more records:

function getPostsFromAPI(argOne, argTwo) {
    return new Promise((resolve, reject) => {
        const apiUrl = `https://www.exampleapi.com/v1/userposts`;

        let results = [];
        let offset = 0;

        const getNextPage = (offset = 0) => {
            apiGet(`${apiUrl}?name=argOne&something=argTwo&offset=${offset}`).then((response) => {
                results = results.concat(response.records);
                if (response.has_more) {
                    getNextPage(response.next_offset);
                } else {
                    resolve(results);
                }
             }).catch(reject);
        }

        getNextPage(0);
    });
}

Note that as a matter of general good practice you should never construct a query string through concatenation or template strings. You should use URLSearchParams.toString() to ensure your query string is properly encoded. You can do so indirectly by creating a new URL:

const url = new URL(`https://www.exampleapi.com/v1/userposts`)

url.searchParams.append("argOne", argOne);
url.searchParams.append("argTwo", argTwo);
url.searchParams.append("offset", offset);

url.toString()

This looks good, and makes a lot more sense - thank you. Would I be correct in assuming that the 'true' condition of the while would be broken by the return? Could I replace it with something like while (offset <= 100)? Thanks for the tip about URLSearchParams as well, I wasn't aware of it — kougami, May 09 '21 at 02:51
Yes, the `while(true)` "breaks" when the function returns, but not the way that `break` (the keyword) would stop it. The flow of execution leaves the function at the point of the `return` statement, any code that appears after the `while` is not executed. You can change the condition to check for `offset <= 100`, that would work fine. — user229044, May 09 '21 at 02:55

Andrew Gillis · Answer 2 · 2021-05-09T03:07:48.210

1

This is a great use case for an async generator.

Would look something like the following

async function* getPostsFromAPI(arg1, arg2) {
  const apiUrl = `https://www.exampleapi.com/v1/userposts`
  let response = { next_offset: 0 };

  do {
    response = await apiGet(`${apiUrl}?name=${arg1}&something=${arg2}&offset=${response.next_offset}`)

    response.items.forEach((item) => {
      yield item
    })

  } while (response.has_more)
}

edited May 09 '21 at 03:07

answered May 09 '21 at 02:56

Andrew Gillis

3,250
2
13
15

I think `response.next_offset || 0` will blow up? As `response` is undefined in the first iteration of the loop. – user229044 May 09 '21 at 02:58

Best way to make a http request and then potentially another one

2 Answers2