0

Background:

I have a multi-processed NodeJS app, that does some heavy CPU processing. The app uses the child_process module, to fork additional subprocesses.

The logic is:

main loop -> fork a Master Worker -> The MasterWorker forks up to N workers -> N workers do processing -> MasterWorker collects results -> posts back to main loop.

N here is always large enough, and greater than the number of CPU cores. Let's say that N=100, # of cores=20, and that a single worker uses 20-25% of a core. This results in 100% CPU usage for a very long time.

Obviously, I can just set N=# of Cores-1, but this limits my throughput.

Setting it to (5*(# of Cores)) -1 is still not perfect, as it depends on what other stuff the machine is doing.

My question:

I would like to keep 1 core totally "free", in order to not bring the machine to its knees while this app runs.

Is there another way that doesn't involve changing the number of worker processes?

I am aware of this: https://stackoverflow.com/a/12522859/1543677, but it's not really doable for the subprocesses the app spawns.

pkExec
  • 1,752
  • 1
  • 20
  • 39

1 Answers1

0

Really simple after all (thanks @AKX).

  1. Install nodeaffinity
  2. At the top of each worker, use:
var nc = require('nodeaffinity');
nc.setAffinity(262143);    //Cores #0-#17, modify to your liking
pkExec
  • 1,752
  • 1
  • 20
  • 39
  • Glad I could help. If you want an approximate round-robin between eg. 5 cores, you could use `setAffinity(1 << (process.pid % 5));` – AKX Mar 02 '23 at 10:27