get data asynchron from Rest api and loading them to mongodb with node.js - using promises

Question

I worked with Javascript not that much and struggle now.

In Short:

I want to retrieve data from a Rest-API and bulk insert them into a collection of a running MongoDB database.

My knowledge so far:

As Javascript is a single-threaded language. So if you want that something is done after other things are finished (mostly async operations), you have to use callbacks. Callback: A Callback is a function A which is passed to a function as a param and must be executed at the end inside that function to guarantee that either A

is executed as the latest action

or

data/objects which had to be retrieved async are possibly available so they could be used inside A

Promises:

Is a relatively new feature introduced to Javascript which could be used to deal with Javascript single-threaded nature? Promises have different states (fulfilled, rejected, pending, settled).

async/await: Are also a new feature to let working with asynchronous functions look more synchronous and to prevent too much nesting which could lead to confusion.

My scenario:

This is a class with a function which uses 'request' to do a REST-Request.

class DataManager {
    static loadSkill(uri) {
        return new Promise((resolve, reject) => {
            const apiEndpoint = 'apiEndpoint';
            request(apiEndpoint + uri, (error, response, body) => {
                if (!error && response.statusCode === 200) {
                    resolve(body)
                } else reject(error);
            });
        });
    }
 ...
}

In the following function, I read the URIs of the data I want to retrieve from a JSON file. The loaded and parsed array is split in arrays of the size of 100 entries because I want to bulk insert 100 objects at once later. In the inner forEach loop, I try to load the skill by using the promise with await, so the function has to be declared with async

function retrieveAndInsertAllSkills() {
    //DBManager.createCollection('skills');
    fs.readFile('./data/uris.json', 'utf8', function readFileCallback(err, data) {
        if (err) {
            console.log(err);
        } else {
            let obj = JSON.parse(data);

            let chunks = chunk(obj.uris, 100);

            chunks.forEach((chunk)=>{
                let skills = [];
                chunk.forEach(async (uri) => {
                    let skill = await Datamanager.loadSkill(uri);
                    skills.push(skill);
                });
                DBManager.insertMany(skills,'skills')
            })
        }
    });
}

At first, I run in an error regarding MongoDB. But I realized that the error was caused by trying to insert empty arrays. So, to be honest, I ran in a similar problem at another problem.

I try to execute a Linux shell command (sha1sum [file]) to get a checksum. The files are large and I

class FileUtil {
    static executeCommand(command, paramsArray) {
        return new Promise((resolve, reject) => {
            let cmd = spawn(command, paramsArray);
            cmd.stdout.on('data', data => {
                resolve(data.toString());
            });
            cmd.stderr.on('data', error => {
                reject(error.toString());
            });
        });
    }
  ...
    static getSha1Sum(file){
        this.executeCommand('sha1sum', [file])
            .then((chkSum) => {
                console.log(chkSum);
                return chSum
            })
            .catch((errorMsg) =>{
                console.log(errorMsg);
                return new Error(errorMsg)});
    };
 ...
}

The following code prints 'undefined' at the console only. The code above prints the desired result.

async function testcheck(file) {
    try {
        let hash = await FileUtil.getSha1Sum(file);
        console.log(hash);
    }
    catch (e) {
        console.log(e);
    }
}
testcheck('/home/coloeus/Downloads/WebStom-2018.2.2.tar.gz');

So my question is: What's wrong with my code. I hope you can give me a hint. I have read a lot about using promises but can't figure out why it isn't working.

[Don't use a `class` with only static methods](https://stackoverflow.com/a/29895235/1048572). JS is not Java! — Bergi, Oct 07 '18 at 20:44
[Don't use `.forEach`](https://stackoverflow.com/q/37576685/1048572). — Bergi, Oct 07 '18 at 20:45
Thank you Bergi. Didn't know the 'problem' with .forEach before. I will have your first link in mind! — coloeus, Oct 15 '18 at 01:14
Your `loadSkill()` also uses the [explicit Promise construction anti-pattern](https://stackoverflow.com/q/23803743/1541563). You should use one of the [recommended libraries](https://www.npmjs.com/package/request#promises--asyncawait) instead so that you don't accidentally mishandle errors. — Patrick Roberts, Mar 28 '19 at 17:41

score 0 · Answer 1 · answered Oct 07 '18 at 19:34

0

The problem is you're calling a function's method without building the class, change this:

async function testcheck(file) {
    try {
        let hash = await FileUtil.getSha1Sum(file);
        console.log(hash);
    }
    catch (e) {
        console.log(e);
    }
}

to this:

async function testcheck(file) {
    try {
        const fileUtil = new FileUtil();
        let hash = await fileUtil.getSha1Sum(file);
        console.log(hash);
    }
    catch (e) {
        console.log(e);
    }
}

And check if the other console.log() inside your promised function call within getSha1Sum(...) gets executed.

answered Oct 07 '18 at 19:34

Gabriel Balsa Cantú

1,954
1
14
11

the function are declared static so no objects/instances are needed to use the method I think. But I tried it by deleting the static declaration at executeCommand() ,getSha1Sum and created a object with new FileUtil() like you suggested but get the same result – coloeus Oct 07 '18 at 19:42
@coloeus can you add a `console.log()` as the first line of `getSha1Sum(file)` like: `getSha1Sum(file){ console.log('got file'); ... } to check that the function is being called? – Gabriel Balsa Cantú Oct 07 '18 at 19:59
Yes, the function is called. – coloeus Oct 15 '18 at 01:13

score 0 · Accepted Answer · answered Oct 07 '18 at 19:40

0

replace the second forEach with for..of, like so:

chunks.forEach(async (chunk)=>{
    let skills = [];
    for (const uri of chunk) skills.push(await Datamanager.loadSkill(uri));
    DBManager.insertMany(skills,'skills');
})

answered Oct 07 '18 at 19:40

Natan Farkash

256
3
4

get data asynchron from Rest api and loading them to mongodb with node.js - using promises

2 Answers2