return object or array in nested javascript functions

Question

I've worked hard to solve a probelm where I can now parse a lot of xml into my js and create an object within the functions that contains all the data from the xml that I want.

What I can't do is get a function to return the object as a result so that I can use it in my wider code.

I've logged to console each exit point of each nested function and worked out where it gets stuck but can't work out how to get the data any further...

Could you have a look and see why it is stuck at the point where I've commented. Many thanks if you choose to help a little!

//now accessing an xml in a file structure
const path = nativeRequire('path')
const fs = nativeRequire('fs');
const xml2js = nativeRequire('xml2js');
const glob = nativeRequire('glob');
//const files = glob.sync(path.join(__dirname, '../../Steinberg/Cubase/Expression Maps/Native Instruments/Symphony Series/*.expressionmap'));
const mapFiles = glob.sync(path.join(__dirname, '../ExpressionMaps/*.expressionmap'));
var artArr = []
var artColor = []
var trackID = '117-4' //test value -  this will come from the TRACK ID sysex stuff


buildMap()   //I need Build Map to populate Objects artArr and artColor and return them so I can use them in other places in the code


function buildMap() {

    mapFiles.forEach(function (mapFile) { //sends mapFile into the parser

        //debug

        //console.log('Selected Tracks  ' + mapFile);  // List out all the expression maps
        //console.log('mapFiles length' + mapFiles.length) // gets the length of list of the files - number of expression maps in directory
        //debug

        var parser = new xml2js.Parser({ explicitArray: false, mergeAttrs: true }); // 'mergeAttrs: true' was essential 

        if (mapFile.includes(trackID)) {


            console.log('Selected Map:  ' + mapFile);


            fs.readFile(mapFile, function (err, data) {

                parser.parseString(data, function (err, result) {

                    let art = result.InstrumentMap.member[1].list.obj;
                    for (let i = 0, len = art.length; i < len; i++) {

                        console.log('Articulation at poition: ' + i + '  ' + art[i].member[1].string.value + '; ' + 'Color Value: ' + art[i].int.value) // articulatins and color value 

                        artArr[i] = art[i].member[1].string.value
                        artColor[i] = art[i].int.value

                        
                    }
                    
                });

                //THE BELOW LOGS TO CONSOLE OK
                console.log('CHECK LOG artArr[0] ' + artArr[0])
                console.log('CHECK LOG artArr[1] ' + artArr[1])
                console.log('CHECK LOG artArr[2] ' + artArr[2])
                console.log('CHECK LOG artArr[3] ' + artArr[3])
                console.log('CHECK LOG artArr[4] ' + artArr[4])
                console.log('CHECK LOG artArr[5] ' + artArr[5])
                console.log('CHECK LOG artArr[6] ' + artArr[6])
                console.log('CHECK LOG artArr[7] ' + artArr[7]) 
            });
            //THE BELOW LOGS TO CONSOLE as UNDEFINED artArr and artColor get stuck here
            console.log('CHECK LOG 2 artArr[0] ' + artArr[0])
            console.log('CHECK LOG 2 artArr[1] ' + artArr[1])
            console.log('CHECK LOG 2 artArr[2] ' + artArr[2])
            console.log('CHECK LOG 2 artArr[3] ' + artArr[3])
            console.log('CHECK LOG 2 artArr[4] ' + artArr[4])
            console.log('CHECK LOG 2 artArr[5] ' + artArr[5])
            console.log('CHECK LOG 2 artArr[6] ' + artArr[6])
            console.log('CHECK LOG 2 artArr[7] ' + artArr[7]) 
        }

    });

}

console.log('Test log a value from artArr ' + artArr[3])  //Needs to return a value but currently returns 'undefined'
console.log('Test log a value from artColor ' + artColor[3])  //Needs to return a value but currently returns 'undefined'

Answer 1

This type of code is a lesson in asynchronous programming. Functions like fs.readFile() and parser.parseString() are asynchronous. That means that when you call them, they initiate their operation and then IMMEDIATELY return and the rest of your code after them runs. This seriously messes up sequential programming unless you write the code correctly for asynchronous programming. It is ONE of the main things to learn when you start coding in nodejs as lots and lots of things are asynchronous in nodejs.

Fortunately, promises can make it easier to write proper asynchronous code, but you still must be aware of it. And, even when using promises, you will still have to be aware of how/when you can use asynchronously retrieved results.

Here's a reworked version of your code:

//now accessing an xml in a file structure
const path = nativeRequire('path');
const fs = nativeRequire('fs');
const xml2js = nativeRequire('xml2js');
const glob = nativeRequire('glob');
const mapFiles = glob.sync(path.join(__dirname, '../ExpressionMaps/*.expressionmap'));
const trackID = '117-4' //test value -  this will come from the TRACK ID sysex stuff

async function buildMap() {
    const artArr = [];
    const artColor = [];
    for (const mapFile of mapFiles) {
        if (mapFile.includes(trackID)) {
            const parser = new xml2js.Parser({
                explicitArray: false,
                mergeAttrs: true
            }); // 'mergeAttrs: true' was essential
            console.log('Selected Map:  ' + mapFile);
            const data = await fs.promises.readFile(mapFile);
            const result = await parser.parseStringPromise(data);
            const art = result.InstrumentMap.member[1].list.obj;
            for (let i = 0, len = art.length; i < len; i++) {
                console.log('Articulation at poition: ' + i + '  ' + art[i].member[1].string.value + '; ' +
                    'Color Value: ' + art[i].int.value) // articulatins and color value
                artArr[i] = art[i].member[1].string.value
                artColor[i] = art[i].int.value
            }
            // once we've run this inner for loop once, we can't
            // run it again because it will overwrite values in 
            // artArr and artColor, so we break the outer for loop
            break;
        }
    }
    return { artArr, artColor };
}


buildMap().then(result => {
    // use your results here
    console.log(result);
}).catch(err => {
    console.log(err);
});

Structural changes to this code:

1. This only uses asynchronous operations that return a promise so we can use await with it to achieve proper sequencing of asynchronous operations in a loop.
2. .forEach() is not promise-aware or asynchronous-friendly so it is pretty much obsolete in modern Javascript program. Use a regular for loop instead, most often a for/of loop which is async/await friendly.
3. Switch to the promise interfaces for both .readFile() and .parseString() so we can use await and make the loop sequence properly.
4. Asynchronously retrieved results should not be stuffed into higher scoped variables and then used elsewhere because the code elsewhere will NEVER know when those variables actually have their proper values thus why you see undefined a lot as your other code is trying to use the values BEFORE they have been filled into the result. Instead, make the results be the resolved value of your returned promise.
5. The caller then must use await or .then() to get access to your asynchronously retrieved results.
6. The second for loop in this code sets the values of artArr[i] and artColor[i]. This assumes that this loop never gets executed more than once (you never get more than one mapFile that .includes(trackID) because if it did, it would be overwriting the prior loop's values in artArr and artColor. To avoid that ever happening, I've forced the loop to only run once by adding a break; to stop the outer loop once, the inner loop has run once. If you intend to run this inner loop more than once, then you need to redesign how you assign to artArr and artColor so they they append to the arrays rather than assign/overwrite previous values.

Other changes:

1. Don't use var any more. Use const if you can or let if you need to modify the contents of the variable. var should be considered obsolete.