What does multiprocessing.Process.init(self) do?

Question

I have been using this link to learn about multiprocessing, but I'm stuck on the second example:

import multiprocessing
import time

class Consumer(multiprocessing.Process):

    def __init__(self, task_queue, result_queue):
        multiprocessing.Process.__init__(self)
        self.task_queue = task_queue
        self.result_queue = result_queue

    def run(self):
        proc_name = self.name
        while True:
            next_task = self.task_queue.get()
            if next_task is None:
                # Poison pill means we should exit
                print '%s: Exiting' % proc_name
                break
            print '%s: %s' % (proc_name, next_task)
            answer = next_task()
            self.result_queue.put(answer)
        return


class Task(object):
    def __init__(self, a, b):
        self.a = a
        self.b = b
    def __call__(self):
        time.sleep(0.1) # pretend to take some time to do our work
        return '%s * %s = %s' % (self.a, self.b, self.a * self.b)
    def __str__(self):
        return '%s * %s' % (self.a, self.b)


if __name__ == '__main__':
    # Establish communication queues
    tasks = multiprocessing.Queue()
    results = multiprocessing.Queue()

    # Start consumers
    num_consumers = multiprocessing.cpu_count() * 2
    print 'Creating %d consumers' % num_consumers
    consumers = [ Consumer(tasks, results)
                  for i in xrange(num_consumers) ]
    for w in consumers:
        w.start()

    # Enqueue jobs
    num_jobs = 10
    for i in xrange(num_jobs):
        tasks.put(Task(i, i))

    # Add a poison pill for each consumer
    for i in xrange(num_consumers):
        tasks.put(None)

    # Start printing results
    while num_jobs:
        result = results.get()
        print 'Result:', result
        num_jobs -= 1

First, could someone please explain exactly what multiprocessing.Process.__init__(self) does? Also I'm not entirely sure how the queue works, and I'm confused how the run method in the Consumer class is executed even though it is never called (explicitly at least...)

If someone could help me walk through the example to get the given output, it would be greatly appreciated.

Maybe [this question](http://stackoverflow.com/questions/625083/python-init-and-self-what-do-they-do) can be helpful to you. I almost propose it as a duplicate of yours. — brandizzi, Jun 25 '12 at 20:22
@brandizzi that question is fundamental question about Python's `self` and `__init__`. The OP's question here is about why `__init__` is calling `multiprocessing.Process.__init__(self)` (it's basically calling it's parent's init) — Alastair McCormack, Dec 06 '16 at 13:32

Russell Borogove · Accepted Answer · 2012-06-25T20:11:27.510

2

When an object of a class is created, its __init__() method gets called automatically to initialize it. In the second example, class Consumer is defined as inheriting from multiprocess.Process. The first thing Consumer does in its initialization is to initialize its base Process, to ensure it's ready to run. Unlike many OO languages, base class __init__() functions aren't automatically invoked in Python.

The run() method is automatically called in the new process when start() is called on the Consumer object in the main process. That's in the multiprocessing docs.

The Queue is how multiple processes communicate. In general, separate processes can't see each other's data, but Queue lets them pass messages back and forth. In this example, the main process is putting several tasks into a queue, and each of the other processes will pull a task from that queue, carry it out, and send the result back on another queue.

edited Jun 25 '12 at 20:11

answered Jun 25 '12 at 20:04

Russell Borogove

18,516
4
43
50

Thanks, I didn't realize that __init__() wasn't automatically called for Process. If Consumer has more methods, would they be automatically called as well? – aensm Jun 25 '12 at 20:09
No, `run()` is special - read the docs. You can explicitly call other methods of the consumer from `run()`, but they won't be called automatically. – Russell Borogove Jun 25 '12 at 20:14
yeah, so why not `super().__init__()`? – CpILL May 18 '20 at 21:55

score 0 · Answer 2 · answered Jun 25 '12 at 20:00

__init__ is a class constructor. Have a look here for some more background.

According to the docs this must be called in subclasses:

If a subclass overrides the constructor, it must make sure it invokes the base class constructor (Process.init()) before doing anything else to the process.

brandizzi · Answer 3 · 2012-06-25T20:20:01.010

Well, this maybe is not the answer you are looking for... but I think is the best one.

The Process object should make a lot of hard stuff when it is created. I do not know what it should do for sure, but I am fairly sure that it should create another operating system process, a pipe for communication etc. etc. By default, all of this stuff is executed in the initialization method (in Python, called __init__()). So if you create a sublcass of Process you should call the parent __init__() method for initializing everything a process need. Since your Consumer class should do everything a Process do, it should have all the stuff a Process has, and the __init__() invocation assures it.

What exactly it does is, actually, irrelevant. That is the whole point of existing method such as __init__(): they execute the hard work for you and you need just to call it. So, we could get deep into the Python source code to see what __init__() does, but for you, who seems be be starting with Python right now, the best answer is that __init__() does a lot of initialization work you do not need to know.

What does multiprocessing.Process.__init__(self) do?

3 Answers3

What does multiprocessing.Process.init(self) do?