0

I am new to node.js and not sure how to handle a background job in a node.js application? Hoping someone could point me in the right direction. Here’s what I am looking for:

I have a node.js application where a user uploads a file. I need to save this file in a specific folder, kick off a background job and send a message to the user that the file was uploaded successfully. The background job will process this file which is in csv format. It will loop thru the file and each row of the file will have couple of links to some other files. It needs to download all those files and then make a call to a 3rd party api, which will process these files one by one. The files could be about 1GB each but I do not need to parse them. The 3rd party api will process these files and send results to the background job. I am thinking I may not need Redis or a queue system in this case as I do not need in-memory processing but correct me if I am wrong. TIA!

mywn9
  • 107
  • 12
  • It sounds like a scenario for a queue Does this answer your question? [Background processes in Node.js](https://stackoverflow.com/questions/31009340/background-processes-in-node-js) – Konrad Oct 20 '22 at 14:54
  • I would use worker threads for that: https://nodejs.org/dist/latest-v16.x/docs/api/worker_threads.html – Marc Oct 20 '22 at 14:54
  • If you don't want queue, run child-process – Konrad Oct 20 '22 at 14:54
  • @Marc - what's the advantage of using worker threads over child process. I have read few articles and it appears that using worker threads is better than child process but I am not very confident. What should I take into consideration when making the decision? Any npm packages that I can use if I go worker threads route? – mywn9 Oct 20 '22 at 22:15
  • @KonradLinkowski - I am not sure I need a queue for the process mentioned above. Do you see a need for it? What's the advantage of using child process over worker threads? Are there any npm packages that I could use for child process? Are there any performance issues with child processes? – mywn9 Oct 20 '22 at 22:20

0 Answers0