0

I have an express.js application. Users request mysite.com/something and they receive a large amount of text. After each request I need to parse data (for example, to find something in thetext and put it into my Redis database). 

app.get('something', function (req, res) {
    var data = //getting data
    res.send(data);
    //parsing data
});

The problem is that parsing data takes too much time and it's synchronous which means users must wait until it finishes to get their data.

Is there any way to parse data in another .js file and allow the users to get results immediately ?

David Hoelzer
  • 15,862
  • 4
  • 48
  • 67
  • put the parsing script in separate file and use child_process.exec to execute the file asynchronously and get back the output in your app. – hassansin Aug 01 '15 at 16:31

1 Answers1

1

If the data amount is large, you should consider using node streams in order to parse it. Use through2 to easily create transform streams. Using streams you will process one data chunk (one object, one csv line, one xml tag) at a time immediately streaming it to response chunk by chunk.

Also you should never use any synchronous code in your app. Everything should be asynchronous.

krl
  • 5,087
  • 4
  • 36
  • 53
  • Thanks can you give me an example? –  Aug 01 '15 at 09:45
  • @Taylor take a look at https://www.npmjs.com/package/through2. it's a good starting point, although streams are tricky. – krl Aug 01 '15 at 09:47
  • It means that if streaming file takes 10 seconds and giving data to user takes 1 second then all users spend 1 second in my site? –  Aug 01 '15 at 09:50
  • @Taylor i'm not sure what you're asking. streaming is better because it processes data one chunk at a time, avoiding loading everything to memory at once, which for large amount data will kill your performance by exhausting available RAM. – krl Aug 01 '15 at 09:54
  • Thanks, For example parsing a 10k Json takes too much times and `res.send(data);` waits until parsing finished. I want to execute another .js file to parsing my data and also users don't wait for parsing –  Aug 01 '15 at 10:08
  • @Taylor can you paste your code somewhere for me to review? – krl Aug 01 '15 at 11:38
  • I need something like this http://stackoverflow.com/questions/18862214/start-another-node-application-using-node-js –  Aug 01 '15 at 12:43
  • @Taylor forking and clustering won't help you much unless you have multiple cores / multiple servers. you still need to be asynchronous, stream user provided data into files and process it using streams if you want to utilize your resources in a proper way. – krl Aug 01 '15 at 13:58