Request/forLoop/Promise Nesting Doll Hell - Node JS

Question

I'm using an API with some limited functionality, and I'm trying to combine several calls to multiple routes to serve the user specific information based on a single keyword search. This creates a lot of separate requests within for loops, which are themselves inside the callbacks of other request results, etc. In the end I'm not being able to wait for all nested requests to be done until I resolve the original Promise and my template engine renders the page and responds to the users get request. Therefore the templating engine is rendering a page with a still empty array of results.

        let discoveredMovies = [];
        let personSearchCast = new Promise ((resolve) => {
            request(UserSearchParamsURL1, {json:true}, (err, res, body) =>{
                if (err) { 
                    console.log(err); 
                    resolve();
                }
                else if(!body.results.length || !body) { 
                    resolve();
                }
                else {
                    for (const person of body) {
                        request(URL2, {json: true}, (err, res2, body2) => {
                            for (const movie of body2) {
                                    request(URL3, {json:true}, (err, res3, body3) => {
                                        movie[CertainParam] = false;

                                        if (Certain Conditions) {
                                            discoveredMovies.push(movie);
                                        }
            
                                        if (Other Conditions) {
                                            discoveredMovies.push(movie);
                                        }
                                        if(More Conditions) {
                                            movie[CertainParam] = true;
                                            Object.assign(movie, {body3info: body3});
                                        }
        
                                        resolve();
                
                                });
                            };
                        });
                    };
                }
            });
        });
        
        personSearchCast.then(() => {
            res.render("partials/foundpeoplemovies.ejs", {movies: discoveredMovies});
        });

As it stands I know that all the Request calls do happen since I can log them out, but the template renders before they can finish. I've tried creating several async await functions and trying to connect them but I just can't wrap my head around it. I also tried using counters to condition the resolve() until all the loops within loops were finished, but that didn't get me too far either.

I know the code is messy, but any ideas how to cleanly do what I'm trying to do? Otherwise I'll have to mess with the user interface and add a series of intermittent steps for the user instead of serving the user all the info in one quick search.

You're not even nesting promises here, you're nesting asynchronous callbacks of `request`. Promisify that function on its own, then use only the promise version - `then`,`Promise.all`,`async`/`await` will easily work then. — Bergi, Jun 19 '21 at 17:02
[The `request` module](https://www.npmjs.com/package/request) is deprecated. You shouldn't use it. (Switching to a module that uses promises by default would make it easier to use `async` and `await`) — Quentin, Jun 19 '21 at 20:40
Thanks to both. I'm still getting the hand of using async functions. I switched from the request module to the request-promise module as well. — Gabe, Jun 21 '21 at 14:55

Mulan · Accepted Answer · 2021-06-19T20:34:29.123

To echo Bergi's comment, you need to "promisify" your request function -

async function requestJson(url) {
  return new Promise((resolve, reject) =>
    request(url, {json: true}, (err, res, body) => {
      if (err)
        reject(err)
      else
        resolve({ res, body })
    })
  )
}

Now you can use Promises as they are intended -

async function searchCast(URL1) {
  const discoveredMovies = []
  const { body: cast } = await requestJson(URL1)
  for (const actor of cast) {
    const { body: movies } = await requestJson(URL2)
    for (const movie of movies) {
      const { body: details } = await requestJSON(URL3)
      if (details[CertainParam] == someCondition)
        discoveredMovies.push({ movie, details })
    }
  }
  return discoveredMovies
}

Here's an example of how to use searchCast in a web form -

myForm.addEventListener("submit", async event => {
  try {
    const searchUrl = ...
    const discoveredMovies = await searchCast(searchUrl)
    console.log(discoveredMovies)
  }
  catch (err) {
    console.error(err.message)
  }
})

Using the for loop as we did above will result in serial queries, ie a movies sub-query must complete before moving onto the next one. You could use Promise.all to run all of the sub-queries in parallel. Another option is to model a Pool to limit/throttle the multiple requests to prevent flooding the remote API. Such a technique is described in this Q&A.

Thanks! Your example helped wrap my head around "promisifying" and then using the request call. As per Quentin's comment I switched to the request-promise module instead of the deprecated request module. It didn't seem to break any of the rest of the code. — Gabe, Jun 21 '21 at 14:53

Request/forLoop/Promise Nesting Doll Hell - Node JS

1 Answers1