Is there a way to use cProfile (or any profiler) with multiprocessing pools?
I'm used to using cProfile like python -m cProfile myscript.py but now that I mainly use multiprocessing that doesn't work. I've been reading the docs and various sites to try get it to work with multiprocessing but I'm a bit stuck - any help appreciated.
This works other than for processes in the threads:
import cProfile, pstats, io
from pstats import SortKey
from multiprocessing import Pool
def threaded_func(x):
for i in range(11111111):
if i%10000000 == 0:
print (x)
pr = cProfile.Profile()
pr.enable()
a = [1,2]
with Pool(processes=2) as pool:
pool.map(threaded_func, a)
threaded_func(3)
pr.disable()
s = io.StringIO()
sortby = SortKey.CUMULATIVE
ps = pstats.Stats(pr, stream=s).sort_stats(sortby)
ps.print_stats()
print (s.getvalue())
i.e., I can see the threaded_func(3) call and all the details of the pooling, but not the calls to threaded_func in the threads:
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
4 0.000 0.000 1.371 0.343 /Library/Python/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3230(run_code)
4 0.000 0.000 1.371 0.343 {built-in method builtins.exec}
1 0.000 0.000 0.735 0.735 <ipython-input-123-ea84061719c4>:14(<module>)
8 0.000 0.000 0.653 0.082 /Library/Python/anaconda3/lib/python3.7/threading.py:534(wait)
8 0.000 0.000 0.653 0.082 /Library/Python/anaconda3/lib/python3.7/threading.py:264(wait)
53 0.653 0.012 0.653 0.012 {method 'acquire' of '_thread.lock' objects}
1 0.000 0.000 0.651 0.651 /Library/Python/anaconda3/lib/python3.7/multiprocessing/pool.py:285(map)
1 0.000 0.000 0.651 0.651 /Library/Python/anaconda3/lib/python3.7/multiprocessing/pool.py:676(get)
1 0.000 0.000 0.651 0.651 /Library/Python/anaconda3/lib/python3.7/multiprocessing/pool.py:673(wait)
1 0.000 0.000 0.635 0.635 <ipython-input-123-ea84061719c4>:17(<module>)
1 0.635 0.635 0.635 0.635 <ipython-input-123-ea84061719c4>:5(threaded_func)
How do I see whats in the threads?
