2

I know how to use multiprocessing to start a separate process and make concurrent data processing.

What I like to know now, is, if I have several main processes (let's say several wsgi processes), how can I create only one separate process for all the main processes ?

Each main process should be able to communicate with the separate process using a queue, but each are started separatly, in a different Python VM.

E.G:

  • process 1 -> start separate process -> talk to separate process
  • process 2 -> notice the separate process is started and don't start another -> talk to separate process
  • process 3 -> notice the separate process is started and don't start another -> talk to separate process

If one process notice the process has died, it can start it again.

Is that possible ? and how ?

If yes suppose it must involve using a PID file.

Bite code
  • 578,959
  • 113
  • 301
  • 329

1 Answers1

1

Yes that is possible. You could use a pidfile or (if you are using Linux) you can use a socket like this: https://stackoverflow.com/a/7758075/639295.

You might also look at using something like redis to do the inter process communication. It has a simple Python api that makes it easy to have multiple processes listening to another.

This is an example of both sides, methods on larger classes. 

def redis_listener(self):
    r = redis.Redis(host='localhost', db=0)
    pubsub = r.pubsub()
    pubsub.psubscribe('a.channel')
    for message in pubsub.listen():
        logging.info('Received message: %s' % (message,))
        self.parse_message(message)


def redis_broadcaster():
    r = redis.Redis(host='localhost', db=0)
    pubsub = r.pubsub()
    pubsub.psubscribe('a.*')
    for message in pubsub.listen():
        if message['pattern'] == None:
            continue
        symbol = message['pattern'].split('.')[1]
        for listener in WATCHERS[symbol]:
            listener.write_message(unicode(message['data']))
Community
  • 1
  • 1
aychedee
  • 24,871
  • 8
  • 79
  • 83
  • The whole purpose is to avoid setting up an additional tool. Otherwise I would just use celery. Thanks for the link, it is helpful. I'm still not certain on how to queue information to that process. – Bite code Aug 17 '12 at 11:21
  • Linux provides a number of methods for IPC. Have a look here: http://tldp.org/LDP/lpg/node7.html, and pick whichever one seems simplest. A named pipe might be the most basic. – aychedee Aug 17 '12 at 11:53