3

I'm using keras to make and test different types of Neural Nets and need data to compare them. I need data on the cpu and memory used during the training and testing. This is in python and as I looked around I found lot of suggestions for psutil. However everything I see seems to grab the current usage.

What is current usage? Like the amount of memory used at that specific moment? How do I use it to get the total cpu and memory used by the entire program, or at least the portion of the code where the NN is training and testing. Thanks for any help!

Windle
  • 1,385
  • 2
  • 14
  • 33
Rex Pan
  • 63
  • 5

1 Answers1

1

psutil is a good recommendation to collect that type of information. If you incorporate this code into your existing keras code, you can collect information about the cpu usage of your process at the time the cpu_times() method is called

import psutil

process = psutil.Process()
print(process.cpu_times())

The meaning of the value returned by cpu_times() is explained here. It is cumulative, so if you want to know how much CPU time your keras code used altogether, just run it before you exit the python script.

To get the memory usage information for your process, at the particular time you make the call to memory_info() you can run this on the same process object we declared before

print(process.memory_info())

The exact meaning of the cpu and memory results depend on what platform you're using. The memory info structure is explained here

A more comprehensive example shows how you could use the Advanced Python Scheduler to take cpu and memory measurements in the background as you run your keras training

import psutil

import time
import os

from apscheduler.schedulers.background import BackgroundScheduler

process = psutil.Process()

def get_info():
    print(process.cpu_times(), process.memory_info())

if __name__ == '__main__':
    scheduler = BackgroundScheduler()
    scheduler.add_job(get_info, 'interval', seconds=3)
    scheduler.start()

    # run the code you want to measure here
    # replace this nonsense loop
    now = time.time()
    finish = now + 60

    while time.time() < finish:
        print("Some progress message: {}".format(time.time()))
        time.sleep(10)
Jeremy Spykerman
  • 78
  • 8
  • What if I want total memory usage like cpu time is that possible to do for memory as well? Or does it only ever grab memory usage at the moment its called? – Rex Pan Apr 10 '19 at 20:39
  • That method only shows current usage. What do you mean by total memory usage? Do you mean the maximum? Do you mean the accumulation of all memory that was allocated by the script? If you want the maximum, you could measure the value at various times, keeping the highest value. If you want accumulated memory allocation, I am not sure how to calculate that. – Jeremy Spykerman Apr 11 '19 at 04:44
  • If you are happy with this answer, can you please accept it? I am trying to build my reputation and it is very difficult to find unanswered questions that I can answer. – Jeremy Spykerman Apr 11 '19 at 05:01
  • Ah is there no way to show accumulative memory usage through the whole program? Max would be nice but the main computational part is part of keras running the model so its not possible to insert anything into it to try to keep track of max usage. Thanks though. – Rex Pan Apr 17 '19 at 18:49
  • I am unaware of how it would be possible to show accumulative memory, but I've enhanced my answer to show how you could take cpu and memory measurements while your keras training is ongoing. – Jeremy Spykerman Apr 18 '19 at 19:20
  • How would I do it if i want to use this to find the max memory usage by checking like ever 1s? I try just having simple if mem < process.memory_info().rss then mem = process.memory_info().rss but doesn't seem to update the variable outside the function. – Rex Pan Apr 24 '19 at 06:59