Trying to wait for list to be populated before iterating through it NodeJS

Question

I have some code that takes an uploaded file, executes it and gets the output. It then compares this output against expected output to check if the script did as expected.

I am now trying to improve this functionality so that an uploaded file will be run several times, each time being checked against a different expected output, or "test case". I then want to push "correct" or "incorrect" onto a results array, so that I can go through that array at the end and check whether there are any "incorrect" (whether the file failed any test case).

I have tried just callbacks within each function.
I have tried using await and async on the getArray as seen below
Using both callbacks and async together.

This is the parent function code that calls for the array to be created, and wants to iterate through it after it has been created.

var resultsArr = await getResults(file.name, files.length, markerDir);
//file.name is the name from the uploaded file object
//files.length is the number of subdirectories (number of testcases to run against)
//markerDir is the str path to where these testcases are stored

if (resultsArr){
   for(var i=0;i<resultsArr.length;i++) {

      if (resultsArr[i] == "incorrect"){                    
         checkForMarkerCb('incorrect'); //Calls back to frontend to
         break;                         //display result on web app
      }

      else if (i+1 == resultsArr.length) {
         checkForMarkerCb('correct');
      }
   }
}

The following is inside the getResults function that is called above

for(var i=1; i<=fileLength; i++) {
   var sampleOut = markerDir + '/test' + i + '/stdout.txt';

   //Grab expected stdout.txt
   var markerOut = fs.readFileSync(sampleOut, 'utf-8', function(err){
      if (err){
         throw err;
      };
   });

   //Run the file and grab the output
   executeFile(filename, function(fileOut){

      //Compare output with sample stdout
      if (markerOut == fileOut){
         resultsArr.push('correct');
      }

      else {
         resultsArr.push('incorrect');
      }
   });
}

//If results array has a response for each testcase
if (resultsArr.length == fileLength) {
   return resultsArr;
}

Implementation of executeFile() as requested:


function executeFile(filename, execFileCb){
   //pathToUpload is a str path to where the upload is stored

   const child = execFile('python', [pathToUpload], (err,stdout,stderr) => {
      if (err) {
         throw err;
      }

      execFileCb(stdout); //Callback with output of file
   });
}

function executeFileAsync(filename) {
    return new Promise(function(resolve,reject){
        executeFile(filename, function(err, data){
            if (err !== null) reject(err);

            else resolve(data);

        });
    });
}

which was called inside getResults() using

var fileOut = await executeFileAsync(filename)

The initial function that calls getResults().
getResults(): which gets the path to each directory and calls pushes the results of comparing outputs onto a results array.
executeFile(): uses 'child_process' to run a file and calls back with the output.

I expect the code to wait for getResults to return with the resultsArr so that the for loop can iterate through and check for any "incorrect". Instead, getResults returns before resultsArr is populated.

Using some logging, I see that the code for checking markerOut == fileOut is executed at the end after the getResults() for loop has already completed. I tried setting up the call to executeFile() to also be an async/await similar to how getResults() is called but still no change.

I may not be using async/callbacks correctly, any help is greatly appreciated.

Can you please show us how you tried to use `async`/`await` in `getResults`? Especially how you did convert `executeFile` so that it returns a promise? — Bergi, Apr 04 '19 at 16:37
@Bergi I declared getResults as "async function getResults(filename, fileLength, markerDir) And when I tried to do this for executeFile, I made it return the output rather than use callbacks. var fileOut = await executeFile(filename) ..similar to how getResults is called in the code above. — BenjyC, Apr 04 '19 at 16:43
Please [edit] your question to show the code. Also show the implementation of `executeFile`, this is what seems to be important. Notice that you cannot simply `return` an asynchronous result, you need to return a promise and later fulfill it. — Bergi, Apr 04 '19 at 18:07
@Bergi made some edits, I was getting confused between promises/callbacks/async and thought using async/await was an alternative to using a promise. Is there a way to fix this with Promises? — BenjyC, Apr 04 '19 at 18:43
You can only use `async`/`await` together with promises, not instead of them. `await` is syntactic sugar around a `.then()` call, but the promise is still there. You need to [make your `executeFile` function return a promise](https://stackoverflow.com/q/22519784/1048572). — Bergi, Apr 04 '19 at 18:44
@Bergi Thank you for that link, Ive tried converting my executeFile function with the callback to the syntax under **"3. Node Style Callback"**.. However Im unsure where I can put my if markerOut == fileOut code. Inside the Promise? Do I save the promise to a var fileOut then do it after? — BenjyC, Apr 04 '19 at 19:19
You'd use `executeFile(filename).then(function(fileOut){ …` or `fileOut = await executeFile(filename);` — Bergi, Apr 04 '19 at 19:23
@Bergi Ive edited the post to show my current implementation, Ive added the function executeFileAsync and I'm calling it how you said, but am getting an *"UnhandledPromiseRejectionWarning"* — BenjyC, Apr 04 '19 at 19:36

score 0 · Accepted Answer · answered Apr 04 '19 at 21:38

Your executeFileAsync function currently calls executeFile with a callback that is expecting two arguments, but executeFile then does call this execFileCb always with only one argument which is interpreted as an error. It also should not use throw in an asynchronous callback.

Instead, merge them into one function:

function executeFile(filename) {
  return new Promise(function(resolve,reject){
    //pathToUpload is a str path to where the upload is stored

    const child = execFile('python', [pathToUpload], (err,stdout,stderr) => {
      if (err) reject(err);
      else resolve(stdout); //Callback with output of file
    });
  });
}

Thank you so much! Finally have it properly waiting for the array to be filled and I now understand Promises. Really appreciate it! — BenjyC, Apr 04 '19 at 23:35

Trying to wait for list to be populated before iterating through it NodeJS

1 Answers1