0

EDIT: Yes, this is technically a question that has been asked before at a high level (why doesn't X get populated when it's outside a loop - asynchronous issue, etc), but the "real" question I was asking was: "what is happening asynchronously in my specific scenario?" For those who see this in the future, the line-reader package uses asynchronous methods, a fact that I was unaware of. The same applies to "forEach" loops - "for of" loops are synchronous, so that is an alternative. Otherwise, use async/await. END OF EDIT

Some background before my question, for context purposes:

I have a file with a bunch of output from a SQL database. The output was generated via several JOIN statements so that all of it could be done in one shot. There are three tables that the data came from, with the columns renamed by AS statements. So if the column came from tableA, it would be named "tableA_someColumnName", if it was from table B, it would be "tableB_someOtherColumnName", etc. The tables are orders, patients, and accounts (nursing homes). One record per patient, one record per nursing home, multiple records for orders. So, lines for different orders may have the same nursing home and patient information in them, like so:

patients_name, accounts_name, orders_order_no
Bob Bobbins, Shady Oaks Retirement, 12345
Bob Bobbins, Shady Oaks Retirement, 54321
Bob Bobbins, Shady Oaks Retirement, 98765

Using Node.js, I am trying to read the SQL output line by line (using line-reader), and insert the unique values into separate MongoDB collections based on the column prefix (orders in the orders collection, patients in the patients collection, etc). Because of the issue where there is duplicate information, I can directly insert the order information into MongoDB, but with patients and accounts, I need to put them in two lists and then deduplicate them after I am done looping through the file so that I only insert unique rows into MongoDB. However, for some reason when I use .push() to add the entries (as objects) to the array that I instantiated, the entries don't persist outside of the .eachLine loop that line-reader uses. I have tried changing let to var and a number of other things, but every time I print my array afterwards, it's empty. If I print it inside the loop, the objects in the array are there and the current object is appended to it. I'm sure it's a scoping issue, but I have no idea what is going on.

Here is my code:

const lineReader = require('line-reader');
const mongoService = require('./mongo.service')
const args = process.argv.slice(2)

const file = args[0];
const db = "test"

//create function to iterate through file and insert entries into MongoDB

function readFileAndInsertInMongo(fileName, dbName) {

    let lineNo = 0;
    let colNums = 0;
    let colHeaders = [];
    let homesList = [];
    let patientsList = [];

    lineReader.eachLine(fileName, function (line, last) {
        console.log("line number is " + lineNo)

        //If lineNo is 0, it must be the header line - get the headers
        if(lineNo === 0) {
            colHeaders = line.split('\t');
            colNums = colHeaders.length;
        } else {
            let ordersObject = {};
            let patientsObject = {};
            let homesObject = {};

            //Split lines on tab character
            lineVals = line.split('\t');

            //For each column header, go through the column name, 
            //get rid of the prefix, and use that as the object key
            //Put column key/value pairs together in 3 different sets of data
            for(i=0;i<=colNums;i++) {
                if(colHeaders[i] !== undefined) {
                    if(colHeaders[i].toString().match(/^claims_|^orders_/)) {
                        ordersObject[colHeaders[i].toString().replace(/^claims_|^orders_/,'')] = lineVals[i];
                    } else if (colHeaders[i].toString().match(/^patients_/)) {
                        patientsObject[colHeaders[i].toString().replace(/^patients_/,'')] = lineVals[i];
                    } else if (colHeaders[i].toString().match(/^accounts_/)) {
                        homesObject[colHeaders[i].toString().replace(/^accounts_/,'')] = lineVals[i];
                    }
                }
            }

            //orders are unique, just push them to MongoDB
            mongoService.addEntry(dbName, "orders", ordersObject);

            //push homesObject to homesList so deduplication can happen later (that part's not coded yet)
            homesList.push(homesObject);

            //HERE IS WHERE I CAN SEE MY ARRAY
            console.log(JSON.stringify(homesList))

            //we want to do the same thing with patients that we did with homes - this presumably doesn't work either
            patientsList.push(patientsObject);
        }

        //ON A RELATED NOTE, THIS ALSO ITERATES JUST FINE, WHICH IS ODD
        lineNo = lineNo + 1;
    });

    //ALTHOUGH I DON'T SEE ANYTHING HERE WHATSOEVER
    console.log(homesList)
}

//Call the function
readFileAndInsertInMongo(file, db)

Thanks in advance.

NellaGnute
  • 225
  • 1
  • 6
  • 15
  • lineReader.eachline is asynchronous. You will have to wait for it to finish in order to see the contents of homelist. I suggest Promises! – MackoyokcaM Dec 19 '18 at 01:31
  • Aha, that's the part I was missing. I figured it was losing the values because there was an asynchronous call in there somewhere, but I wasn't sure where it was. Thanks @MackoyokcaM ! – NellaGnute Dec 19 '18 at 02:29

0 Answers0