1

I have a node program that does a lot of heavy synchronous work. The work that needs to be done could easily be split into several parts. I would like to utilize all processor cores on my machine for this. Is this possible?

Form the docs on child processes and clusters I see no obvious solution. Child processes seems to be focused on running external programs and clusters only work for incoming http connections (or have I misunderstood that?).

I have a simple function var output = fn(input) and would just like to run it several times, spread all the calls across the cores on my machine and provide the result in a callback. Can that be done?

Ludwig Magnusson
  • 13,964
  • 10
  • 38
  • 53

1 Answers1

1

Yes, child processes and clusters are the way to do that. There are a couple of ways of implementing a solution to your problem.

  1. Your server creates a queue and manages that queue. Whenever you need to call your function, you will drop it into the queue. You will then process the queue N items at a time, where N equals the number of your cores. When you start processing, you will spawn a child process, probably either using spawn or exec, with the argument being another standalone Node.js script, along with any additional parameters (it's just a command line call, basically). Inside that script you will do your work, and emit the result back to the server. The worker is then freed up.

  2. You can create a dedicated server with cluster, where all it will do is run your function. With the cluster module, you can (once again) create N number of other workers, and delegate work to these wokers.

Now this may seem like a lot of work, and it is. And for that reason you should use an existing library as this is a, for the most part, a solve problem at this point. I really like redis-based queues, so if you're interested in that see this answer for some queue recommendations.

Community
  • 1
  • 1
Yuri Zarubin
  • 11,439
  • 4
  • 30
  • 33
  • Ok so basically what you are saying is that my assumptions about child processes and clusters are correct and if I want to just execute a function to do this I need to put it in a standalone script that I or make a server that just performs the function call and get the response through a local http request? – Ludwig Magnusson Sep 11 '15 at 09:16
  • That's right, node.js is a single threaded event loop, and javascript has no threading library. If you want to use more cores, you need additional loops, hence additional processes via child processes. – Yuri Zarubin Sep 11 '15 at 14:20
  • I am aware of the event loop. I just find it strange that when cluster capabilities are added to node, they are strictly tied to the http server. – Ludwig Magnusson Sep 11 '15 at 14:36
  • They aren't tied to the http server. The cluster module just spawns processes, it has no context of http. I used the word "server" in the general sense. – Yuri Zarubin Sep 11 '15 at 14:40
  • Then I have misunderstood the cluster documentation. Will make another deep dive into it. – Ludwig Magnusson Sep 12 '15 at 09:44
  • Don't know why this has been accepted as a valid answer. I'm struggling with the same thing. I need to run a synchronous process thousands of times and am trying to use the clustering method, but all examples I can find are with the http server and not running a function directly – Oliver Wagner Sep 12 '22 at 11:33