2

The Question

Why isn't it possible to send more than messages (strings) to forked subprocesses in NodeJS? This answer simply states that is isn't possible but provides no explanation.

How this question came about

I need to send database connection to subprocesses to avoid connecting to those databases multiple times.

My program connects to four different services: Redis, OrientDB, Postgres, and ElasticSearch.

I'm doing some heavy lifting, and need to use multiple processes and have all of the processes connect to these services. I've made an object to hold all of these clients:

var clients = {
    redisClient: /* redis connection */
    orientClient: /* orientdb connection */
    pgClient: /* postgres connection */
    elasticClient: /* elasticsearch connection */
};

But when I send the clients to subprocesses:

var child = cp.fork(__dirname + '/singleCrawler.js');
child.send(clients);

I get the following error:

TypeError: Converting circular structure to JSON 
    at Object.stringify (native)
    at ChildProcess.target.send (child_process.js:451:23)

I'm not the only one having this problem.

My solution to the problem

I'm going to spawn another process that handles the services. The other processes will communicate with this process to interact with the services.

Community
  • 1
  • 1
Jay
  • 18,959
  • 11
  • 53
  • 72
  • Why wouldn't you want multiple connections if you're using multiple forks? – WiredPrairie Nov 10 '13 at 21:23
  • Normally I would (client pooling and the like), but the orient driver for node isn't quite on par with the others & doesn't support this. The additional information was probably unnecessary noise for the question asked. – Jay Nov 10 '13 at 21:40
  • Doesn't support what? Connections from multiple processes from the same machine/server? – WiredPrairie Nov 10 '13 at 21:52
  • I'm writing a bit of code that has a high frequency of collisions. Orient can handle multiple connections at once, but if something hits the same vertex at the same time it starts to break down. [node-orientdb](https://github.com/gabipetrovay/node-orientdb) doesn't seem to have a clean solution for this yet whereas [node-postgres](https://github.com/brianc/node-postgres#client-pooling) supports client pooling. Orient itself [does support this](https://github.com/orientechnologies/orientdb/wiki/Performance-Tuning#network-connection-pool), just not the nodejs driver. – Jay Nov 10 '13 at 22:34

2 Answers2

1

The reason why you can't pass any arbitrary javascript object to a child process is because it runs a completely separate V8 process that does not share any memory with the parent process.

In your case, I would open a separate set of connections for each process. Since you won't have hundreds of node process, this won't be a huge burden on your dbs and it will make your code simpler.

(Off topic)

There is actually one exception to what you can pass to another process: you can pass a file descriptor:

child.send(message, [sendHandle])

The sendHandle option to child.send() is for sending a TCP server or socket object to another process. The child will receive the object as its second argument to the message event.

This is used by the cluster module. That's not very useful in your case but I thought I'd mention that :)

Community
  • 1
  • 1
Laurent Perrin
  • 14,671
  • 5
  • 50
  • 49
1

One reason why it's not possible is because it isn't trivial: if you were to have one connection to Redis shared among several processes, how would concurrent reads/writes be handled?

At least on Unix-type OS'es, you can pass file descriptors between processes, but that's not going to solve the concurrency issue. It's also very low-level and each Node driver (Redis, OrientDB, Postgres, ElasticSearch) would somehow have to support being passed a file handle (which they don't, and I wonder if it's even possible).

Your solution with one process that's managing all the database connections is a good solution, although I wonder why a few extra database connections is such a big issue.

robertklep
  • 198,204
  • 35
  • 394
  • 381
  • Thanks for the explanation. The additional information on databases was noise to the question - the reason I can't open multiple connections is that the orient driver for node isn't quite on par with the other three and can't handle client pooling gracefully. I've upvoted this answer and accepted the "first" response - Both answers provided were right on point however. – Jay Nov 10 '13 at 21:44