This can be done elegantly with Ray, a system that allows you to easily parallelize and distribute your Python code.
To parallelize your example, you'd need to define your functions with the @ray.remote decorator
, and then invoke them with .remote
.
import ray
ray.init()
# Define functions you want to execute in parallel using
# the ray.remote decorator.
@ray.remote
def func1():
#does something
@ray.remote
def func2():
#does something
# Execute func1 and func2 in parallel.
ray.get([func1.remote(), func2.remote()])
If func1()
and func2()
return results, you need to rewrite the code as follows:
ret_id1 = func1.remote()
ret_id2 = func1.remote()
ret1, ret2 = ray.get([ret_id1, ret_id2])
There are a number of advantages of using Ray over the multiprocessing module. In particular, the same code will run on a single machine as well as on a cluster of machines. For more advantages of Ray see this related post.