Asynchronous Programming with Nodejs - Iterating through an array of links, opening them and saving them to JSON once done

Question

I have an array of tracking links (about 30), which I want to open piece by piece and find out the real URLs hidden behind them. Once that's done, I want to save the 'real' URLs to a JSON file.

The URLs look something like this before they have been "checked": https://www.trackinglink.com/1

and something like this afterwards: https://www.amazon.com/

I have solved the "uncovering" of the tracking links using request and it works. However, what I can't manage to get to work, is waiting with writing the JSON file until all the URLs have been "requested"/checked.

I know that the solution involves Async/Await or Promises, but I can't get it to work in node. For someone more experienced, this is probably a matter of a few minutes.

The concept of asynchronous programming is pretty much new to me, but I have spent my fair share of hours researching it. I think I have difficulties transferring the knowledge out there to my specific problem.

I'd really appreciate the help. Cheers!

const request = require('request');
const fs = require('fs'); 
let listWithRealUrls = [];

function grabAndSaveRealUrls() {

    let Urls = ['https://www.trackinglink/1', 'https://www.trackinglink/2', 'https://www.trackinglink/3']

    for (const Url of Urls) {
        request.get(Url, function () { 
            let realUrl = this.uri.href; 
            listWithRealUrls.push(realUrl)
        });

    }

    fs.writeFile('data.json', JSON.stringify(listWithRealUrls), function (err) {
        if(err) {
            console.log(err);
        } else {
            console.log('success');
        }
    })

}

grabAndSaveRealUrls();

You also don't learn swimming by watching other people swim. Wrap the `request.get` calls into a promise, then push the promise (synchronously) instead of it's result (which arrives asynchronously). The you can use `Promise.all` on that array, and in it's `.then` handler you can access all the results, which you can write to a file then. — Jonas Wilms, Nov 19 '19 at 17:49
i believe you can avoid using promises t`by adding a if statement in the callback of **request.get** checking if all urls are tested, and if so write the file — gui3, Nov 19 '19 at 17:51
@gui3 sure, and then do the same thing with two, three, ... loops, nested functions etc. *This doesn't work out in the long run*. Better do do things the correct way from the beginning (but it might help understanding how `Promise.all` works ...) — Jonas Wilms, Nov 19 '19 at 17:52
That worked just fine @JonasWilms, thank you! Is there any way to post my solution here, so anyone facing similar issues can just take a quick look? Unfortunately my question has been marked as a duplicate (which is wide generalization in my oppinion, but hey...). — Tom Doe, Nov 20 '19 at 11:26
@TomDoe sure! Glad you figured it out! ... Make sure to link the [duplicate](https://stackoverflow.com/questions/14220321/how-do-i-return-the-response-from-an-asynchronous-call) prominently for future visitors ... — Jonas Wilms, Nov 20 '19 at 11:28

Tom Doe · Accepted Answer · 2019-11-20T16:07:49.237

Thanks to Jonas' comment and the previously linked 'duplicate question', I managed to solve this. I am sure there are more elegant ways, but here's how I did it:

const request = require('request');
const fs = require('fs'); 
let listHoldingPromises = [];

function grabAndSaveRealUrls() {
    let Urls = ['https://www.trackinglink.com/1', 'https://www.trackinglink.com/2', 'https://www.trackinglink.com/3']

    for (const Url of Urls) {


      let promise = new Promise(function(resolve, reject){

        request.get(Url, function () { 
            let realUrl = this.uri.href; 
            resolve(realUrl);
        });
      }) 

      listHoldingPromises.push(promise);

    }
}

grabAndSaveRealUrls();


Promise.all(listHoldingPromises).then(values => {
    fs.writeFile('data.json', JSON.stringify(values), function (err) {
        if(err) {
            console.log(err);
        } else {
            console.log('success');
        }
    })
  });

Asynchronous Programming with Nodejs - Iterating through an array of links, opening them and saving them to JSON once done

1 Answers1