0

How would you recommend I go about fetching multiple files (in particular, CSV files) and merging them into a single variable? I've tried using the following and I'm pretty sure I got something about the promise syntax/logic wrong, as that is an area I'm fairly new to, but it may be another problem. Here is my current attempt:

fetcher(specURL).then(async (spec) => {
    let rawData;
    if (typeof spec.dataURL === "string") {
        rawData = await fetcher(spec.dataURL);
    } else if (typeof spec.dataURL === "array") {
        rawData = {};
        await spec.dataURL.forEach(async (url, i) => {
            rawData[i] = await fetcher(url);
        });
    }
    console.log(rawData);
}

As you can see I use a spec (specification) json which articulates which URLs to fetch. If it is an array I would like to fetch multiple files into an array.

I'm using my fetcher function, which is the following (any recommendations to improving it are appreciated too!

async function fetcher(url, type) {
let fetched = await fetch(url);
fetched = await fetched.text();
if (type == undefined) {
    type = url.split(".").pop();
}
if (type == "json") {
    fetched = JSON.parse(fetched);
} else if (type == "csv") {
    fetched = Papa.parse(fetched, { header: true }).data;
}
return fetched;

}

Noam I.
  • 73
  • 8
  • what is the contents of specUrl? – Muthukumaran Sep 19 '21 at 09:39
  • @Muthukumaran The specURL contains multiple things, but the relevant part is that it contains the following array that keyed to dataURL : "dataURL": [ "User input/02-09-2021.csv", "User input/04-09-2021.csv", "User input/03-09-2021.csv" ] – Noam I. Sep 19 '21 at 09:42
  • Is that a Json something like a single { "dataURL":["file1.csv","file2.csv"]} – Muthukumaran Sep 19 '21 at 09:44
  • specURL is the location of a JSON, which is fetched by the fetcher() function, and then put as an object into the spec variable in the asynchronous function. – Noam I. Sep 19 '21 at 09:46
  • Is there a reason you want to fetch the data synchronously. The better approach might be to submit an array of URLs to the fetcher function, make an array of fetch promises by `map`ping over the array for each URL, await them the promises to return with `Promise.all`, and then process the data. – Andy Sep 19 '21 at 09:47
  • @Andy could you please elaborate? Generally speaking, it seems like what you are suggesting will do the trick just as well. I was thinking of fetching the data synchronously for performance reasons. – Noam I. Sep 19 '21 at 09:49
  • It's not clear how you're using `type` as you're not passing it in as an argument anywhere @noambugot1 – Andy Sep 19 '21 at 10:15
  • I'm using it in an "if" in the fetcher function – Noam I. Sep 19 '21 at 10:55

0 Answers0