0

I am trying to make a simple server that has some heavy load processing. I am using cluster module to take this heavy load to threads and let Node make it's magic. The problem is: as I don't know how to build a thread pool, I'm afraid I might run into some trouble with pid limit.

The question: Provided that I can't change process identifier OS limit, how can I create a thread with Node that does not die (a thread that waits for a message, processes it and then waits for another message) WITHOUT using busy waiting (I want them to block while waiting for a new request)?

bellati
  • 82
  • 1
  • 9
  • What PID limit will you hit? If you're creating so many processes or threads, you hit a max cap, you've created too many. As you've noticed, Node doesn't make this type of work simple. Node is best for events IO, not crunching. There are far more natural platforms for threading than Node. – WiredPrairie Nov 22 '13 at 11:47
  • pid is unique among an instance of a machine. If you spawn, if I'm not mistaken, 32k processes (even tough some of them are dead), you will end up hitting the maximum pid number (which will give you an error). – bellati Nov 22 '13 at 14:25
  • 1
    You are mistaken. They get reused. If you have 32K active processes, your machine is overworked and spending too much time context switching between threads/processes rather than doing actual work. http://stackoverflow.com/questions/11323410/linux-pid-recycling – WiredPrairie Nov 22 '13 at 14:31
  • Also, like I said, NodeJS isn't a very good match for "heavy load processing". – WiredPrairie Nov 22 '13 at 14:32
  • nice, I didn't know about the recycling. Good work! It solves my problem. – bellati Nov 22 '13 at 14:39
  • I'd like to know if there's any way to block a thread to wait on an event and, as soon as it processed it, it would wait for another call. Is this possible without using busy waiting? – bellati Nov 22 '13 at 14:41
  • No, they're called events in Node. You've picked the wrong platform. – WiredPrairie Nov 22 '13 at 15:34

1 Answers1

1

I can't get what you're asking for, but node cluster meant to spawn a constant number of workers (usually one per CPU core) to enable multithreading processing of your request.

Each worker works in a single thread, consuming one and only one pid. All workers share the same TCP connections, allowing requests to be distributed between them. Each worker process all request, dispatched to id, asynchronously (all at the same time) in its single thread.

Node.js designed to utilize all resources of a single CPU core by processing all incoming requests asynchronously, meaning you don't need more than numCPUs workers to utilize all your resources.

So, I cant't understand your problem with pid limit.

If you have problems configuring your cluster, see this answer and this blog post.

Summarizing my answer, properly configured cluster consists of require('os').cpus().length worker processes handling requests and one master process watching them and respawning dead ones.

Community
  • 1
  • 1
Leonid Beschastny
  • 50,364
  • 10
  • 118
  • 122
  • I see. So you are saying that if I am running Node in a 4 core machine, The maximum number of heavyweight threads is 4? – bellati Nov 22 '13 at 14:26
  • Yes. It's possible to spawn more than `numCPUs` workers, but you don't need to. But as bellati said, Node is not a bad choice for processing heavy load requests, though it's a perfect choice for processing a vast number of light ones. – Leonid Beschastny Nov 22 '13 at 14:56
  • So, if your goal is to dispatch heavy operations to workers, then you should consider using non-node.js ones. – Leonid Beschastny Nov 22 '13 at 15:06