1

I want to process array of length around 100 000, without putting too much load on CPU. I researched about streams and stumbled upon highlandjs, but i am unable to make it work.

I also tried using promises and processing in chunks but still it is putting very much load on CPU, program can be slow if needed but should not put load on CPU

James Z
  • 12,209
  • 10
  • 24
  • 44
Sumit Bhanushali
  • 33
  • 8
  • What is the reason for low load on the CPU? Server responsiveness to other requests? Something related to your hosting plan? What? That would help us know what things to suggest. Also, please show code for what the processing is that you are doing? As this question stands now, there's no way to answer it. It takes a certain amount of CPU cycles to do your processing on the array. Nothing you do (other than rewriting the actual code that processes the array to be more efficient) changes that. – jfriend00 Jul 06 '19 at 21:30
  • As written, this is an [XY problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem) where you're describing your attempted method of solving a problem and not describing the underlying actual problem. We can help you better if you describe the actual underlying problem. – jfriend00 Jul 06 '19 at 21:43
  • Upon high load on CPU, server stops responding to other requests and sometimes crashes, so i want to prevent that and about implementation I haven't posted because that contains basic array processing like finding element from other array, using it, etc – Sumit Bhanushali Jul 07 '19 at 03:45

1 Answers1

3

With node.js which runs your Javascript as single threaded, if you want your server to be maximally responsive to incoming requests, then you need to remove any CPU intensive code from the main http server process. This means doing CPU intensive work in some other process.

There are a bunch of different approaches to doing this:

  1. Use the child_process module to launch another nodejs app that is purposeful built for doing your CPU intensive work.
  2. Cluster your app so that you have N different processes that can do both CPU intensive work and handle requests.
  3. Create a work queue and a number of worker processes that will handle the CPU intensive work.
  4. Use the newer Worker Threads to move CPU intensive work to separate node.js threads (requires node v12+ for stable, non-experimental version of threads).

If you don't do this CPU intensive work very often, then #1 is probably simplest.

If you need scale for other reasons (like handling lots of incoming requests) and you don't do the CPU intensive stuff very often #2.

If you do the CPU intensive stuff pretty regularly and you want incoming request processing to always have the highest priority and you're willing to allow the CPU intensive stuff to take longer, then #3 (work queue) or #4 (threads) is probably the best and you can tune the number of workers to optimize your result.

jfriend00
  • 683,504
  • 96
  • 985
  • 979