0

This node.js app reads text file and checks the each line has * charset. And if it contains * charset, modify without * charset.

[Input]

*2
*54
*74
*82
*148
*180
*184

[Output]

2
54
74
82
148
180
184

And here's my code

fs.readFile('data.txt', async function (err, data) {
    if (err) throw err;
    let array = data.toString().split("\n");
    for (i in array) {
      if (array[i].charAt(0) === '*') {
        // console.log('YES');
        console.log(`Now Processing : ${array[i]} | ${array.length - i -1} items left`);
        //
        // SOME JOBS
        //

        let newValue = array.map(function (line) {
          return line.replace('*', '');;
        }).join("\n")
        // let newValue = array[i].replace('*', '');
        // console.log(newValue)
        // // newValue = newValue + '\n';

        await fs.writeFile('data.txt', newValue, 'utf-8', async function (err, data) {
          if (err) throw err;
          // console.log('Done!');
        })

      } else {
        console.log(`${array[i]} Already Captured`)
      }
    }
  });

The problem is when I put small data in to the app. It works find. But When I tried large data, the app crashed. When I tried like (0 to 3852218) it crashed. How can I solve this problem?

[Error Message]

<--- Last few GCs --->

[39840:000001F9D690E460]    64836 ms: Mark-sweep 2047.1 (2072.6) -> 2047.0 (2073.4) MB, 19.7 / 0.0 ms  (average mu = 0.154, current mu = 0.001) allocation failure scavenge might not succeed
[39840:000001F9D690E460]    64849 ms: Mark-sweep 2047.6 (2073.9) -> 2047.3 (2073.7) MB, 12.3 / 0.0 ms  (average mu = 0.129, current mu = 0.091) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 00007FF69879404D]
Security context: 0x018506f3bbb1 <JSObject>
    1: join [00000148A0B52EB9](this=0x01346739cb39 <JSArray[38164]>,0x001adca1dfd9 <String[#1]\: \n>)
    2: /* anonymous */ [000001CD18B49CE1] [C:\Users\super\Downloads\Projects\dcinside-recommended-post-capture\app.js:55] [bytecode=0000010F22DB4791 offset=213](this=0x00d17da3d5c9 <JSGlobal Object>,0x01b192bc01b1 <null>,0x01cd18b49d19 <Uint8Array map = 00...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200207.011709.39840.0.001.json
Node.js report completed
writingdeveloper
  • 984
  • 2
  • 21
  • 46
  • Can you quantify "large data", how big is the text file you are trying to read? – sinanspd Feb 06 '20 at 16:29
  • 1
    Yep, you need to learn about Node's [streams](https://www.freecodecamp.org/news/node-js-streams-everything-you-need-to-know-c9141306be93/) – Jeremy Thille Feb 06 '20 at 16:31
  • 1
    Just process a line at a time. The [`readline`](https://nodejs.org/api/readline.html#readline_readline) module is your friend. – spender Feb 06 '20 at 16:31
  • ...and [this](https://stackoverflow.com/a/33419220/14357) answer should see you good for the writing part. – spender Feb 06 '20 at 16:33

0 Answers0