Writing data to multiple files in for loop using Node.js

Question

In my current project, I am reading data through API and I would like to have multiple files for each device (I have around 100 devices) and for each device, I would like to create .csv file, write multiple rows to the file (using node csv-write-stream package) and close the file. Similarly, I would like to do the same operation for each 100 devices. I tried the following code but I am getting an error that:

writing after end

   var csvWriter = require('csv-write-stream');
   var writer = csvWriter({ headers: ["TimeStamp", "Gas","gw_temp","rh","abrasion","sensor_temp"]});
      for(let j=0;j<data.length;j++){
      // Create file for each devices
       await writer.pipe(fs.createWriteStream(data[j][0] + '.csv',{flags:"a"}));
       // Get data through API
       await axios.get([API CALL URL].join('/'), getConfig)
        .then(async (response) => {
          let res = await response.data;
           if(res.length!=0){
           for (let i = 0; i < res.length; i++) {
              // Write data to csv file
              await  writer.write(res[i]);
           }
          }
        })
        .catch((error) => {
          if (error.response) {
            console.log("Error in response", error.message);
          } else {
            console.log("Other error", error.message);
          }
        });
        await writer.end(); 
        await console.log("File writing done",data[j][0]);
       }
      }

It looks like that I am missing a minor thing but I couldn't able to recognize it.

I also refer this link: "Error: write after end" with csv-write-stream

Updated If I remove the catch block then it throws the following error:

(node:13264) UnhandledPromiseRejectionWarning: Error: write after end at writeAfterEnd (_stream_writable.js:236:12) at CsvWriteStream.Writable.write (_stream_writable.js:287:5)

Why are you combining `async`/`await` and `.then`/`.catch`? Similarly, on the line `let res = await response.data;`, is `response.data` *really* a promise? — T.J. Crowder, Aug 08 '18 at 12:04
`async/await` is to get the promise execution and `catch` is to catch the error from `API`. — Saurabh Chauhan, Aug 08 '18 at 12:05
The point is that you use one of the other in general, you don't combine them. `async`/`await` is syntax for consuming promises with synchronous-looking code, you don't normally need `.then` or `.catch` (in an `async` function, `.then` => `await`, `.catch` => `try`/`catch`). Suggest some [reading up](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function). :-) — T.J. Crowder, Aug 08 '18 at 12:06
Finally: What's `writer` in this code? Are you sure you can reuse it like that? (Given the error, I'm thinking you probably can't.) — T.J. Crowder, Aug 08 '18 at 12:07
I have updated the code. Could you please check it again? Yes I need to reuse the `writer` for each devices. — Saurabh Chauhan, Aug 08 '18 at 12:09
@T.J.Crowder: If I will remove the `catch` block then It will through error as mentioned in the updated question. — Saurabh Chauhan, Aug 08 '18 at 12:13
I never said to *remove* the error handler. As I said above: In an `async` function, `.catch` => `try`/`catch`. Please, again, take a moment to read through [the MDN page on `async` functions](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function). — T.J. Crowder, Aug 08 '18 at 12:15
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/177635/discussion-between-saurabh-chauhan-and-t-j-crowder). — Saurabh Chauhan, Aug 08 '18 at 12:18

score 0 · Accepted Answer · answered Aug 08 '18 at 12:52

Finally, I figured out the solution. I need to initialize the writer variable every time before writing data to file as it ends with the writer.end(). I am sharing the code:

  // Get sensor measurement data for each devices
  for(let j=0;j<data.length;j++){
  // Intialize the writer everytime before creating a file
    var writer = csvWriter({ headers: ["TimeStamp", "Gas","gw_temp","rh","abrasion","sensor_temp"]});
    await writer.pipe(fs.createWriteStream(data[j][0] + '.csv'));
    await axios.get([API CALL URL].join('/'), getConfig)
    .then(async (response) => {
      let res = await response.data;
      //console.log(res.length);
      if(res.length!=0){
       for (let i = 0; i < res.length; i++) {
         await writer.write(res[i]);
       }
      }
      await console.log("File writing done",data[j][0]);
      await writer.end();

    })
    .catch((error) => {
      if (error.response) {
        console.log("Error in response", error.message);
      } else {
        console.log("Other error", error.message);
      }
    });
  }

Writing data to multiple files in for loop using Node.js

1 Answers1