Is there any way to print out the gpu memory usage of a python program while it is running?

Question

I am trying to optimise my GPU memory usage for my python program and on task manager I can see that it stays low for a while, and then at a certain point it shoots upwards. I am not sure at what point in the program this is though, so I was wondering if there was a way to print it to the screen while the program is running to help me locate the cause of the sudden increase?

As you can see from the screenshot, my memory and CPU usage go up at around the same time too, so any commments on how to print those would be appreciated even if the former is not possible.

In `pytorch` you can use `torch.cuda.memory_allocated()` function — Girish Hegde, Aug 27 '20 at 14:21

score 1 · Accepted Answer · answered Aug 27 '20 at 14:21

1

It looks like you are on windows, which is more challenging to do this for. On Linux, you can just throw some !nvidia-smi commands in your code and it will give you a readout of the GPU usage information. Similar commands can be done for CPU and memory usage on Linux.

For windows this link seems to be what you are looking for https://stackoverflow.com/a/2468983/12288338

answered Aug 27 '20 at 14:21

alwaysmvp45

437
4
8

Thanks, I think that the link you give only covers main memory and not gpu, so anything on that would be appreciated too. – Arkleseisure Aug 27 '20 at 14:56

Is there any way to print out the gpu memory usage of a python program while it is running?

1 Answers1