9

I'm trying to run cProfile.runctx() on each process in a multiprocessing pool, to get an idea of what the multiprocessing bottlenecks are in my source. Here is a simplified example of what I'm trying to do:

from multiprocessing import Pool
import cProfile

def square(i):
    return i*i

def square_wrapper(i):
    cProfile.runctx("result = square(i)",
        globals(), locals(), "file_"+str(i))
    # NameError happens here - 'result' is not defined.
    return result

if __name__ == "__main__":
    pool = Pool(8)
    results = pool.map_async(square_wrapper, range(15)).get(99999)
    print results

Unfortunately, trying to execute "result = square(i)" in the profiler does not affect 'result' in the scope it was called from. How can I accomplish what I am trying to do here?

Fragsworth
  • 33,919
  • 27
  • 84
  • 97
  • Does this answer your question? [Python multiprocess profiling](https://stackoverflow.com/questions/11041683/python-multiprocess-profiling) – Chris Withers Jan 17 '23 at 15:14

1 Answers1

8

Try this:

def square_wrapper(i):
    result = [None]
    cProfile.runctx("result[0] = square(i)", globals(), locals(), "file_%d" % i)
    return result[0]
Cat Plus Plus
  • 125,936
  • 27
  • 200
  • 224
  • 2
    +1; that worked, but seems fairly hacky. Can you explain why it worked? – Fragsworth Sep 12 '09 at 11:13
  • 2
    I guess 'result = square(i)' just created a new reference, in cProfile.runctx scope (or wherever it exec'd the code), leaving old one intact. Using "global result" before runctx, and "global result; result = square(i)" (or "globals()['result'] = square(i)") inside works too. – Cat Plus Plus Sep 12 '09 at 11:23