Finding Memory Usage, CPU utilization, Execution time for running a python script

Question

I am using python, and suppose i had some code as below

example.py

import os, psutil
import MySQLdb as mdb

conn = mdb.connect(user='root', passwd='redhat', db='File_Data', host='localhost', charset="utf8")
file_path = "/home/local/user/Module/File_processing/part-file.txt"
p = psutil.Process(os.getpid())
cursor = conn.cursor()

for line in file_open:
    result = line.split('\t')
    query = "insert into PerformaceReport (campaignID, keywordID, keyword, avgPosition)"
    query += " VALUES (%s,%s,'%s',%s)%(result[0],result[1],result[2],result[3])"
    cursor.execute( query )
conn.commit()

print p.get_cpu_times()
print p.get_memory_percent()

The above code reads data from text file and saves to database and its working fine

I am running the file with the command python path/to/the/example.py

Now what i am trying to do is finding/capturing the below three values/information

1. The total execution time taken by the python script(example.py) to read and saves data to the database
2. CPU utilization when the python script(example.py) has run(executed)
3. Memory usage when the python script(example.py) has run(executed)

I already googled a lot and found some information here, here and here, each explanation has different approach and i am really confused and unable to understand the approach on how to capture the above three values when the script example.py has run so asked in SO

Can anyone please provide me an working example(code) that suits/captures the above values for my need i mentioned above, so that i can extend further to my further needs

Edited

Process Followed

1 . For finding execution time

Run the example.py file with the command as below

  python -m cProfile -o file.profile example.py 

2 . For finding CPU Usage

Executing the top command in a terminal after the example.py has runned

Result:

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  1354 root      20   0  138m  17m 7200 S  2.0  0.6   2:40.50 Xorg                                             
  2672 shivakri  20   0  581m  19m  12m S  1.3  0.6   0:07.60 gnome-terminal                                   
  2587 shivakri  20   0  987m 168m  33m S  1.0  5.6   5:25.26 firefox                                          
  2780 shivakri  20   0 1233m  51m  16m S  1.0  1.7   0:16.51 gedit                                            
  2163 shivakri  20   0  821m 135m  43m S  0.7  4.5   4:58.74 chrome                                           
  1938 shivakri  20   0  567m  15m  11m S  0.3  0.5   0:06.64 metacity  
  ........
  ........

Now in the above result, how/where can i find the CPU/Memory utilization values, whether i need to take the values %CPU %MEM from the line that contains gnome-terminal ? (Because we are running the file through terminal ?)

Aslo top is used for finding total cpu utilization processes that are running, but we need to find the cpu utilization for only example.py script when it is executed/running, i mean how much CPU, Memory is utilized when we run the file(example.py)

Finally what my intention is when i run the python script (example.py) the values such as execution time,CPU utilization taken for running the script,and Memory used for running the script has to be recorded/stored some where(In to a text file) or something

Answer 1

For time profiling

1. cd into the dir that contains example.py (lets call this exampledir).
2. run python -m cProfile -o example.profile example.py
3. download RunSnake and unpack it anywhere
4. cd into the dir where you unpacked RunSnake
5. run python runsnake.py exampledir/example.profile

For CPU Profiling

1. Use psutil
   1. create a new psutil.Process with myProcess = psutil.Process(os.getpid())
   2. call myProcess.get_memory_info() or myProcess.get_ext_memory_info() or myProcess.get_memory_percent() as required

For memory profiling

1. install meliae with easy_install or pip
2. Add the following lines of code to the top of example.py:

---

from meliae import scanner # [[source](http://www.vrplumber.com/programming/runsnakerun/)]
scanner.dump_all_objects( filename ) # you can pass a file-handle if you prefer

---

3. run runsnakemem fpath, where fpath is the path to the file that you used in the code above.

This should get you a visual memory profiler similar to What you got with RunSnakeRun.

Hope this helps

Answer 2

Here is an informative blog post if anyone still searching anything regarding this requirements:

Monitoring memory usage of a running Python program