How to create a process inside a process using multiprocessing in python?

Question

I have a function that takes 2 minutes to run.
I call it twice in my program so i use map from multiprocessing to have them run in parallel.

Now i want to reduce the time even more .

In this function i have 8 operations that can be run in parallel.

So ,what i can do to have 2 main processes run in parallel and inside each one i have another 8 child processes run in parallel too.?

Possible duplicate of [Is it possible to run function in a subprocess without threading or writing a separate file/script.](https://stackoverflow.com/questions/2046603/is-it-possible-to-run-function-in-a-subprocess-without-threading-or-writing-a-se) — Ricardo Magalhães Cruz, Nov 03 '18 at 20:44

mucka · Accepted Answer · 2017-04-13T10:30:23.137

I am not sure why you need such setup but anyway:

import _thread
import time

def operation(a, b, s):
    print("Starting operation {} in process {}".format(a,b))
    time.sleep(s)
    print("Finished operation {} in process {}".format(a,b))

def process(n):
    _thread.start_new_thread(operation, (1, n, 1))
    _thread.start_new_thread(operation, (2, n, 2))
    _thread.start_new_thread(operation, (3, n, 1))
    _thread.start_new_thread(operation, (4, n, 2))

_thread.start_new_thread(process, (1,))
_thread.start_new_thread(process, (2,))
time.sleep(3)

Works in Python 3.4.

EDIT: As suggested, that multiprocessing may be faster, below same example with multiprocessing

from multiprocessing.pool import Pool as PoolParent
from multiprocessing import Process, Pool
import time

class NoDaemonProcess(Process):
    def _get_daemon(self):
        return False
    def _set_daemon(self, value):
        pass
    daemon = property(_get_daemon, _set_daemon)

class MyPool(PoolParent):
    Process = NoDaemonProcess

def operation(a):
    print("Starting operation {} in process {}".format(a[0],a[1]))
    time.sleep(a[2])
    print("Finished operation {} in process {}".format(a[0],a[1]))

def process(n):
    p = Pool(4)
    p.map(operation, [(1,n,1), (2,n,2), (3,n,1), (4,n,2)])

p = MyPool(2)
p.map(process, [1,2])
time.sleep(3)

@roganjosh I have provided additional example using multiprocessing as you suggested, it is just simpler using `_thread` because of internal locks of `multiprocessing` and requires simple hack to get this work. — mucka, Apr 13 '17 at 10:32
Understood. However, the OP's aim was to improve execution speed; threading cannot achieve that due to the GIL so it leaves no option other than to use multiprocessing. — roganjosh, Apr 13 '17 at 10:33
@roganjosh ,thank u i'm going to test that and see the results — KAFFEL, Apr 13 '17 at 11:10

How to create a process inside a process using multiprocessing in python?

1 Answers1