0

Brief Intro

I am developing server side application in NodeJS which include one module which read data from csv file and perform some task.

In CSV file data could be in Millions lines but at the same time I am having limited memory available.

To perform a task I am using kue (creating a job per line)

currently I am creating jobs as soon as CSV reader give me dataand kue is keeping that job in memory and will not leave until it finish (So till the CSV reader finish reading file there is millions of job exists in memory).

So as a solution What I want to do is I read first few hundreds lines create jobs let wait till it finish then read another hundreds.

Here waiting is more important I can not use any module which allow me to read file line by line because in that case it will keep returning me liens and I will keep creating jobs.

I want control on reader such that when I get desired hundreds (ex 100) I will wait till all 100 will finish then ask reader to return next hundreds (ex next 100).

Conclusion

I want to know how I can read few lines from a csv file then process it and then read remaining (Same is pagination).

Code will look something like below

const smartReader = required('snartReader');
smartReader.read(100,(data,next) => {
   performJobs(data); // wait until it finish then
   next(); // it will read next 100    });

Declaration

I am not an experienced person in NodeJS and this is my first application.

Jageen
  • 6,345
  • 2
  • 37
  • 56
  • 3
    Node supports [streams](https://nodejs.org/api/stream.html#stream_stream), you could also use [Readline](https://nodejs.org/api/readline.html#readline_readline) – Rafael May 11 '19 at 08:40
  • Ya in that case how can I control stream I mean like if I want it to wait in half a way till i proceed remaining lines and continue and again wait for the same till I reach EOF. – Jageen May 11 '19 at 08:43
  • Possible duplicate of [Read a file one line at a time in node.js?](https://stackoverflow.com/questions/6156501/read-a-file-one-line-at-a-time-in-node-js) – Rafael May 11 '19 at 08:44
  • As @Rafael pointed out streams are great and you don’t have to manage them, they manage themselves. See this SO about streaming large JSON data, https://stackoverflow.com/questions/55863329/nodejs-save-json-file/55864396#55864396 – Chris Adams May 11 '19 at 08:47
  • @ChrisAdams problem is not because i am reading large file problem is because i mean creating job as soon as i read a line. – Jageen May 11 '19 at 08:53
  • Solution i thought was i will read few lines only at first then finish jobs then read next few lines – Jageen May 11 '19 at 08:54
  • Have you taken a look at the [csv package](https://www.npmjs.com/package/csv) in npm? It's designed to work with large csv files. – RichS May 11 '19 at 09:13
  • @RichS its not about reading large file its about creating jobs in kue – Jageen May 11 '19 at 09:17
  • Well, it's both actually. You want to stream a large part of your csv. the csv package is comprehensive. Not just reading csv files. – RichS May 11 '19 at 19:52

0 Answers0