1

Similar to this question on profiling multiprocessing code...

I need to profile my codebase, and I'd planned to use cProfile. This program is essentially a genetic algorithm that runs an evaluation function in parallel. The kicker is that this evaluation function externally calls another Python program via the command line.

Is it possible to use cProfile to profile the overall run? Basically, I'd intended to start the profiling inside my worker function (cumulative time, pcalls, memory overhead, etc.), however I'm concerned that the external programs will not be visible to the profiler.

Community
  • 1
  • 1
erik
  • 3,810
  • 6
  • 32
  • 63

1 Answers1

1

You are correct that the external programs will not be "visible" to the profiler. The cProfile module adds software hooks to your code that trigger various counters on events like function calls or returns. You will be able to see your call to subprocess.call() or whatever you're using to execute the external commands, but you will not see any detail about the external command's execution. You will be able to see how long the call blocked, so you'll get an idea of how long the external command takes to execute, but you won't get the profiling details of them.

That being said, there's no reason you can't add the cProfile module to the called script as well, and you'll get profile results for both your main script and the called external script. They just won't be in the same file.

skrrgwasme
  • 9,358
  • 11
  • 54
  • 84
  • That's actually what I ended up doing. I simply instrumented my external call to cProfile to only look at that module. – erik Oct 08 '14 at 02:05