How to handle asynchronicity in Nodejs?

Question

I have this case where I have a lot of data I need to sort into different buckets. If there's already a bucket for a certain data type then push it in there, but if the encountered data type is new I need to make a new bucket for it which requires a DB query. This works fine in theory. But, the problem I have is the DB queries need to happen before the loop continues so future data can be inserted into that bucket. How can I pause a loop until a function finishes?

I've tried the loop and callback approach and I get the control flow I'm looking for, but the data is too large and I stack overflow. Is there another way solve this control flow?

EDIT: Added code with promises like people suggested. It actually works well, but just stops at data point 8428. As far as I can tell that data point isn't special and all the buckets have been created at that point so its not creating a new bucket. It just stops. Any ideas? Maybe its how I call next() and I'm missing a case?

async function sortData(data, images, i) {
    let uuidList = [];
    //This loop will wait for each next() to pass the next iteration
    for (var j = 0; j < data.length; ++j) { 
        await new Promise(next=> {
            let pos = uuidList.map(function(e) { return e.uuid; }).indexOf(data[j].uuid);
            if(pos < 0){
                Instance_Type.findInstance(data[j]._instance_type, function(err, instance){
                    /* add new bucket here */
                    if(itemsProcessed == images.length) {
                        processBuckets(uuidList, threshold, function(){
                            console.log("Bucket finished");
                            dataPackage.push({
                                imageName: images[i].name,
                                uuidList: uuidList,
                            });
                            ++itemsProcessed;
                            if(itemsProcessed == images.length){
                                console.log("Rendering!");
                                res.render('data', {
                                    dataPackage: dataPackage
                                });
                            }
                        });
                    }
                    next();
                });
            } else {
                /*push to existing bucket here*/
                if(itemsProcessed == images.length) {
                    processBuckets(uuidList, threshold, function(){
                        console.log("Bucket finished");
                        dataPackage.push({
                            imageName: images[i].name,
                            uuidList: uuidList,
                        });
                        ++itemsProcessed;
                        console.log(itemsProcessed);
                        if(itemsProcessed == images.length){
                            console.log("Rendering!");
                            res.render('data', {
                                dataPackage: dataPackage
                            });
                        }
                    });
                }
                next();
            }

        })       
    }
}

Can you post the current code that works as desired (except for the error), we can probably find a way to fix it? — CertainPerformance, Jun 14 '18 at 20:47
Make it `async`, then `await` the creation. And you usually don't get a StackOverflow with callbacks. Please show us your code. — Jonas Wilms, Jun 14 '18 at 20:48
What's the error you get exactly? As @CertainPerformance suggested, please post the current code. You must be using a library to run the queries that either returns a promise and/or accepts a callback function with which you're supposed to continue the execution of your code asynchronously. — Alessandro, Jun 14 '18 at 20:54
I tried with promises as people suggested and had a weird issue. Added code as requested. — Ashton Spina, Jun 14 '18 at 21:36

score 1 · Answer 1 · answered Jun 14 '18 at 20:51

1

Have you tried async await. For async functions you can await for the promise to return in your case database calls and once that resolves you can move to next.Please go through and other reference it will help your cause. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/await

answered Jun 14 '18 at 20:51

PJAutomator

344
3
12

Thanks, I tried async functions, worked well, but had a weird issue. – Ashton Spina Jun 14 '18 at 21:49

score 1 · Answer 2 · answered Jun 14 '18 at 20:57

1

If I'm understanding your problem correctly, I think you could simply achieve it by using Promises? And maybe combine it with an interval that you can set/clear to control when you want your loop to halt/continue, if you want to run stuff periodically.

IF not familiar with Promises I'd suggest to check:

answered Jun 14 '18 at 20:57

heldic

367
1
11

Thanks, I tried Promises, worked well, but had a weird issue. – Ashton Spina Jun 14 '18 at 21:49
What happened? Maybe we can debug this for you. – heldic Jun 14 '18 at 21:53

score 1 · Answer 3 · answered Jun 14 '18 at 21:12

As far as loop pause is concerned, if you have Node 7.6 or above.

let myData = ["a", "b", "c"];

//TODO: Use try catch
async function findMyBucket() {
   for (let x = 0; x < myData.length; x += 1) {
      var resp = await checkCreateAndInsertInBucket(myData[x]); //Loop pauses
      console.log(resp);
   }
}

function checkCreateAndInsertInBucket(data) {
   return new Promise((resolve, reject) => {
      //DB Logic here with a promise or callback
      resolve(true); //or reject
   });
}

findMyBucket();

How to handle asynchronicity in Nodejs?

3 Answers3