How can i asynchronously and incrementally fetch data inside a map function

Question

I'm trying to incrementally fetch episodes from an array, so the user experience would look like this:

A user clicks a button, shows a loading indicator, loads the first episode fetched inside a div, but keeps showing the loading indicator below that first episode, keeps doing this until the map is over and all the episodes are fetched and the loading indicator is not shown anymore.

But apparently, this does not appear to be working as planned, I'm suspecting that async-await does not work as expected inside a map.

I'm wondering if something like rxjs would help in this use case(I have no idea how it works, but I know it is used in cases where the data is streaming)

  const [isLoading, setIsLoading] = useState(false);
  const [error, setError] = useState('');
  const [data, setData] = useState([]);

  const handleClick = async () => {
    try {
      season.episodes.map(async (episode) => {
        setIsLoading(true);
        const links = await publicQueryClientApi(
          `${process.env.NEXT_PUBLIC_SERVER_URL}/download/${episode.slug}`
        );
        setIsLoading(false);
        setData((prev) => [...prev, { episode, links }]);
        setError('');
      });
    } catch (error) {
      setIsLoading(false);
      setData([]);
      setError(error.message);
    }
  };

map isnt executed sequentially, its executed parallely for "optimization reasons" (?) . If you want something like this then use a foreach or for loop https://stackoverflow.com/a/47369317/2757519 — Dev Man, Oct 31 '21 at 15:47
I'm not sure if scala's `foreach` works the same a javascript, but I just found out for this use case, using `for of` would work as expected as stated here: https://stackoverflow.com/a/37576787/7808470 Thanks @DevMan, this inspired me to find an answer. — Yassine Bridi, Oct 31 '21 at 15:56
@DevMan there is *no* parallel execution in javascript, as it is singlethreaded. What you probably mean, if the callback function of `map` is not awaited if it's async, you you have multipe pending promises, which may resolve in any order. — derpirscher, Oct 31 '21 at 15:59

score 2 · Answer 1 · answered Oct 31 '21 at 16:03

async function always returns a Promise.

You can try something like this:

const promises = season.episodes.map(async episode => {
  const links = await fetchLinks()
  // do other stuff
  return links
})

const links = await Promise.all(promises)

And the other thing is, you only use map when you want a copy of the array but modified (using a function you pass to map that will be executed on every array item). From what I can see your return value from map function is not saved anywhere.

Why not just use recursive function, that will fetch each episode, and call itself again to fetch another for as long as there are "unfetched" episodes? (Don't forget to mark episode as "fetched" each time, otherwise you will make an infinite loop).

derpirscher · Accepted Answer · 2021-10-31T16:11:00.580

array.map does not await its callback, if it's async. So the result of

[...].map(async x => {})

is an array of pending promises. If you want to add your episode one by one, you should use a

try {
  for (let episode of season.episodes) {
    setIsLoading(true);
    const links = await publicQueryClientApi(
      `${process.env.NEXT_PUBLIC_SERVER_URL}/download/${episode.slug}`
    );
    setIsLoading(false);
    setData((prev) => [...prev, { episode, links }]);
    setError('');
  }
} catch (e) {
  ...
}

As your code is already inside a async function, this should work without any further modifications.

Another possibility is to use Promise.all(), this will resolve once all of the promises in the array are resolved and return their results in an array

try {
  setIsLoading(true);
  let allLinks = await Promise.all(season.episodes.map(e => {
    return publicQueryClientApi(...);
  });
  setIsLoading(false);

  setData(season.episodes.map((e,i) => ({
     episode: e,
     links: alllinks[i]
  }));
  setError('');
} catch (e) {
  ...
}

This is what I did, for of works as expected. – Yassine Bridi Oct 31 '21 at 16:09 — Yassine Bridi, Oct 31 '21 at 16:09

How can i asynchronously and incrementally fetch data inside a map function

2 Answers2