0

I'm using Node.js, and I have to update a .csv with the data of another .csv which I'll have access later. For example, if I have these two rows in the .csv:

18/10/2019,19/10/2019 (Key) ,135.45 --> First .csv 18/10/2019,19/10/2019 (Key) ,150 ---> Second .csv

I want the updated row to be:

18/10/2019,19/10/2019 (Matched key) ,150

So, I'm searching for a npm package or another tool which allow me to define the key I'm searching in the first .csv and update it with the values I have in the second.

The problem is, I can't really save the first .csv in JSON format, for example, because the amount of data is so big that the process of saving the data and parsing it afterwards would probably cause me huge memory issues (I'm talking about 14GB csvs more or less).

So... Should I use a database? Any other ideas?

First post in here, I hope I was clear about what I was asking, and sorry for some grammar issues, English is not my native language...

Thanks in advance!

Volterra90
  • 1
  • 1
  • You should use stream instead of loading an entire CSV file in memory. Have a look at below question. https://stackoverflow.com/questions/50469355/how-to-load-very-large-csv-files-in-nodejs – Akash Salunkhe Oct 29 '19 at 13:40
  • Thanks for the answer! I know I can use stream to read a .csv, my problem is how to compare the two .csv. I could read the first .csv and find the row I need to update according to the data I have in the second one, but my question is, is that efficient? I'd have to read a 14GB .csv to find one row and start again at the beginning to find the next one and update it, and so on... I'd say that'd probably take ages, am I wrong? – Volterra90 Oct 29 '19 at 14:15
  • You can open two streams of two different csv files and then compare them and then output it into single csv file. Have a look at how to use multiple streams. https://stackoverflow.com/questions/16431163/concatenate-two-or-n-streams or you can extract keywords from first csv and save those keywords and then compare it with other csv file but you still have to use stream if size of file is too large. – Akash Salunkhe Oct 30 '19 at 08:39

0 Answers0