0

I am dealing with a file 16GB with 100M + rows in it. As a first step, I want to process each line with business logic and apply business rules. After that, I want to store it in the MySql database. For the first step, I implemented a node script, and its working fine with small files, but its giving File size (14241096214) is greater than possible Buffer: 2147483647 bytes and Cannot create a string longer than 0x3fffffe7 characters when I run with big files. Any suggestions on how to process this BIG file and import it into MySQL? Thanks in advance!

Nagesh
  • 121
  • 11
  • 1
    16GB is not very big, it's small.... – Mitch Wheat Jun 15 '20 at 02:51
  • 1
    Typically an sql export can be processed piece by piece, whether by query (delimiting by `;`) or by CSV (delimiting by newlines). Since your file is larger than 2^32 bytes, you need to process the file iteratively. I would try `fs#createReadStream` instead of `fs#readFile` – Rogue Jun 15 '20 at 03:57
  • 2
    You have a problem on the 1st step - i.e. when you process the file using your "node script". It seems that you try to load the whole file into memory for this processing, but you cannot. I see 2 possible variants: 1) Rewrite your "node scripts" and process your file by chunks which does not oversize the buffer 2) Import your data into (temporary) MySQL table then process the data on MySQL side and save it into working tables. I cannot say what is preferred in your particular case... – Akina Jun 15 '20 at 04:44
  • I agree with @Akina. Load the whole thing into a load table first, then do your logic in the database. It will also help a great deal with debugging and traceability. – TomC Jun 15 '20 at 05:23

0 Answers0