Wait for unknown amount of Observables to complete

Question

I have an API I need to pull some data from, but this data is paginated and I have no way to get around it. I don't know how many pages there are, but I can know when I'm at the last one. I want to make a function that gets every page one by one.

This is the page interface:

interface PaginatedResult<T> {
  count: number;
  next: string;
  prev: string;
  results: T[];
}

When I'm at the last page, the next attribute is null, so I essentially want to do something like this:

let data: Data[] = []
let url: string = API_ENDPOINT

while (url!== null) {
  this.http.get(url).subscribe((page: PaginatedResult<Data>) => {
    data = data.concat(page.results)
    url = page.next
  })
}

However, this obviously doesn't work, because it just makes requests at full tilt until one of them sets the url variable to null, and this just freezes the browser. I looked up ways to chain subscriptions together, but I haven't found a way to combine an indeterminate amount of subscriptions together. There is no way to know how many pages are there, I can only get one page at a time and that's all the info I have.

Essentially, the problem is I need to wait for each page to arrive before seeing if I should request the next one.

Any ideas?

As a little note, although this question has a pretty similar title, it doesn't describe the same problem, as in that question the user knows how many requests they need to make, but want to make them in series. In my question I don't know how many requests I have to make, and I want to do them in series.

score 3 · Answer 1 · answered Jun 07 '23 at 12:05

You can use the expand operator to recursively request the new data until the next property of the response is null.

this.http.get(url).pipe(
  expand((page) => page.next ? this.http.get(page.next) : EMPTY),
).subscribe((page) => (data = data.concat(page.results)));

score 2 · Answer 2 · answered Jun 07 '23 at 14:30

While the other answers work perfectly fine, they will make the data available right away as soon as each request finishes.

If you want to wait for all requests to finish and only then emit the value, you can use a recursive function utilizing the switchMap operator.

const loadData = (x: PaginatedResult<Data>): Observable<PaginatedResult<Data>> => {
  if (!x.next) {
    return of(x);
  }

  return this.http.get(x.next).pipe(
    map(page => ({...page, results: x.results.concat(page.results) })),
    switchMap(loadData)
  );
};

this.$data = loadData({next: 'API_ENDPOINT', results: []} as PaginatedResult<Data>).pipe(map(x => x.results));

score 1 · Accepted Answer · answered Jun 07 '23 at 11:45

Not familiar about rxjs but consider doing something like below:

let data: Data[] = []
let url: string = API_ENDPOINT

const handler = (page: PaginatedResult<Data>) => {
  data = data.concat(page.results)
  if (page.next) {
      this.http.get(page.next).subscribe(handler);
  } else {
    // Got everything
  }
}

this.http.get(url).subscribe(handler);

This should get them one at a time sequentially.

Wait for unknown amount of Observables to complete

3 Answers3