0

I have a python code which on profiling with cProfile always gives the following data.

`299149 function calls (294133 primitive calls) in 39.049 seconds

 Ordered by: internal time

 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    2   36.532   18.266   36.532   18.266 {posix.waitpid}
   39    1.769    0.045    1.769    0.045 {method 'query' of '_mysql.connection' objects}
    1    0.120    0.120    0.167    0.167 connections.py:62(__init__)
    2    0.083    0.042    0.083    0.042 {method 'rollback' of '_mysql.connection' objects}`

Now the only way I suspect the code to be blocking on posix.waitpid is because of the following function.

    process = Popen(command, stdout=PIPE)
    return process.communicate()

This is explained fairly well in python subprocess communicate() block .

The method executeCommand is called from a lot of places in the code .

and in another place where it iterates over a list of URLs and executes the curl command. This can be very bad performance bottleneck as this will block for the entire subprocesses to finish.

My question is why cProfile output shows me only 2 in nCalls column although the call to executeCommand function is made multiple times in the code.

If the culprit is what I suspect to be , is there a workaround for this problem.

Community
  • 1
  • 1
station
  • 6,715
  • 14
  • 55
  • 89
  • I see at least two questions: 1. Why `ncalls` is `2` for `posix.waitpid()` and 2. How to improve performance of the code that collects output from multiple external commands, urls. Here's [how to run subprocesses in parallel](http://stackoverflow.com/a/23616229/4279), here's [how to make multiple network connections concurrently](http://stackoverflow.com/a/20722204/4279). – jfs Oct 22 '14 at 06:42
  • more examples:, [fetch multiple urls using multiprocessing.pool.ThreadPool](http://stackoverflow.com/a/23284285/4279), or [Brute force basic http authorization using httplib and multiprocessing](https://gist.github.com/zed/0a8860f4f9a824561b51). See also [Are urllib2 and httplib thread safe?](http://stackoverflow.com/q/5825151/4279). – jfs Oct 22 '14 at 06:51
  • So does this mean that anytime only two sequential batch processescan run in python. – station Oct 22 '14 at 11:58
  • 1
    no. To answer the first question, [provide MCVE](http://stackoverflow.com/help/mcve). – jfs Oct 22 '14 at 11:59
  • 2
    to answer your question: *"My question is why cProfile output shows me only 2 in nCalls column although the call to executeCommand function is made multiple times"* provide a *complete* *minimal* code example that demonstrates the issue: I should be able to run `python -mcProfile your_script.py` and see ncalls equals `2` for `posix.waitpid` while `your_script.py` runs `executeCommand()` more than two times. – jfs Oct 22 '14 at 12:17

0 Answers0