Iterating the creation of objects with asynchonous javascript map method data

Question

In an async IIFE at the bottom of this javascript, you'll see that I'm trying to: 1) read a JSON file, 2) get multiple RSS feed URLs from that data, 3) pull and parse the data from those feeds, and create an object with that data, so I can 4) write that pulled RSS data object to a JSON file. Everything for #1 and #2 is fine. I'm able to pull data from multiple RSS feeds in #3 (and log it to console), and I'm comfortable handling #4 when I get to that point later.

My problem is that, at the end of step #3, within the const parseFeed function, I am trying to create and push an object for that iteration of rssJSONValsArr.map() in the IIFE and it's not working. The rssFeedDataArr result is empty. Even though I am able to console.log those values, I can't create and push the new object I need in order to reach step #4. My creating of a similar object in #2 works fine, so I think it's the map I have to use within const parseFeed to pull the RSS data (using the rss-parser npm package) which is making object creation not work in step #3. How do I get rssFeedOject to work with the map data?

import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import Parser from 'rss-parser';
const parser = new Parser();

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);

const feedsJSON = path.join(__dirname, 'rss-feeds-test.json');

const rssJSONValsArr = [];
const rssFeedDataArr = [];

const pullValues = (feedObject, i) => {
  const url = feedObject.feed.url;
  const jsonValsObject = {
    url: url,
  };
  rssJSONValsArr.push(jsonValsObject);
};

const parseFeed = async (url) => {
  try {
    const feed = await parser.parseURL(url);

    feed.items.forEach((item) => {
      console.log(`title: ${item.title}`); // correct
    });

    const rssFeedOject = {
      title: item.title,
    };
    rssFeedDataArr.push(rssFeedOject);
  } catch (err) {
    console.log(`parseFeed() ERROR : ${err}`);
  }
};

(async () => {
  try {
    console.log('1: read feeds JSON file');
    const feedsFileArr = await fs.promises.readFile(feedsJSON, {
      encoding: 'utf-8',
    });
    const jsonObj = JSON.parse(feedsFileArr);

    console.log('2: get feed URLs');
    jsonObj.slice(0, 30).map(async (feedObject, i) => {
      await pullValues(feedObject, i);
    });
    console.log('rssJSONValsArr: ', rssJSONValsArr); // correct

    console.log('3: pull data from rss feeds');
    rssJSONValsArr.map(async (feedItem, i) => {
      await parseFeed(feedItem.url, i);
    });
    console.log('rssFeedDataArr: ', rssFeedDataArr); // empty !!!

    // console.log('4: write rss data to JSON file');
    // await fs.promises.writeFile(
    //   `${__dirname}/rss-bulk.json`,
    //   JSON.stringify(rssFeedDataArr)
    // );

    console.log('5: Done!');
  } catch (err) {
    console.log(`IIFE CATCH ERROR : ${err}`);
  }
})();

Example JSON file with two RSS feed URLs:

[
  {
    "feed": {
      "details": {
        "name": "nodejs"
      },
      "url": "https://news.google.com/rss/search?q=nodejs"
    }
  },
  {
    "feed": {
      "details": {
        "name": "rss-parser"
      },
      "url": "https://news.google.com/rss/search?q=rss-parser"
    }
  }
]

Any and all help appreciated. Thanks

Answer 1

The problem is you are printing rssFeedDataArr right after the .map call, which, like stated on the comments, is being incorrectly used, since you are not using the returned value, forEach would be the way to go here. For every value in rssJSONValsArr you are calling an anonymous and async function which in turn awaits for parseFeed, so you are basically creating a Promise in each iteration, but obviously those promises are resolved after your print statement is executed. You need to wait for all of those promises to be resolved before printing rssFeedDataArr. One way to do that, since you are creating a bunch of promises which can be run in parallel is to use Promise.all, like this:

await Promise.all(
  rssJSONValsArr.map(async (feedItem, i) => {
    await parseFeed(feedItem.url, i);
  });
)

and you we can simplify it even more and return the promise created by parseFeed directly:

await Promise.all(
  rssJSONValsArr.map((feedItem, i) => parseFeed(feedItem.url, i))
)

And in this case the right method is map and not forEach

In the case of rssJSONValsArr it works because the call to pullValues is being resolved instantly, it doesnt run asynchronously, even when its declared as async, there is not await inside the function definition.