0

There is an array with more than two million records, I want to build a new array from it, but the function does not work. Even after a twenty minute wait nothing happens. The code freezes when assigning the variable result. What might be the problem?

const arr1 = fs.readFileSync('./csv1.csv', 'utf-8')
    .toString()
    .split('\n')
    .map(row => row.split(';'))

const arr2 = iconv.encode(
    iconv.decode(fs.readFileSync('./csv2.csv'), "win1251"), "utf8")
    .toString()
    .split('\n')
    .map(row => row.split(';'))

const concatArrays = arr1.concat(arr2);

const uniqNumber = _.uniq(concatArrays.map(el => parseInt(el[0])))

const result = uniqNumber.map(el => {
    return [parseInt(el), concatArrays.filter(([number]) => el === parseInt(number))]
});
ravil
  • 1
  • 1
  • what errors do you get? – bill.gates Jun 09 '20 at 13:42
  • Node.js has a limited size for buffers (depending on a number of factors related to your hardware). If you file is particularly big you will get an error like "RangeError: Attempt to allocate Buffer larger than maximum size". If that's the case, you might be better off using streams (fs.createReadStream()) and processing the data in a streaming fashion – Luciano Mammino Jun 09 '20 at 13:48
  • 1
    FYI, your first four lines of code are making four copies of the data. Three of them will eventually get garbage collected, but for processing large amounts of data, this makes for very high peak memory usage. You should probably take Luciano's suggestion and process the data incrementally using a stream rather than reading it all into memory and processing it all at once. – jfriend00 Jun 09 '20 at 14:15
  • This one might be related: https://stackoverflow.com/questions/29766868/node-js-read-big-file-with-fs-readfilesync – Luciano Mammino Jun 10 '20 at 09:03
  • createReadStream together with event-stream solved the problem – ravil Jun 10 '20 at 14:23

0 Answers0