5

I've just started an instance on a Google Compute Engine with 2 GPUs (Nvidia Tesla K80). And straight away after the start, I can see via nvidia-smi that one of them is already fully utilized.

I've checked a list of running processes and there is nothing running at all. Does it mean that Google has rented out that same GPU to someone else?

enter image description here

It's all running on this machine:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.5 LTS
Release:    16.04
Codename:   xenial

enter image description here

Vit D
  • 193
  • 1
  • 7
  • 25
  • I need to make sure what is happening on the boot. Can u provide logs from the console output? what are you running on the startup? May you are running a process that uses GPU on startup. Also, the GPU is just using by you and not rented or use by another user. – Milad Tabrizi Oct 17 '18 at 22:45
  • There was nothing running, as I started a clear new machine on GCE. Literally booted for the first time. – Vit D Oct 18 '18 at 12:22

2 Answers2

6

Enabling "persistence mode" with nvidia-smi -pm 1 might solve the problem. ECC in combination with non persistence mode can lead to 100% GPU utilization.

Alternatively you can disable ECC with nvidia-smi -e 0.

Note: I'm not sure if the performance actually is worse. I can remember that I was able to train ML model despite the 100% GPU utilization but I don't know if it was slower.

Stefan
  • 1,246
  • 1
  • 9
  • 13
  • It does not seem to affect performance but it can be misleading if your monitoring your VM remotely. – ma3oun Jan 16 '20 at 15:05
0

I would like to suggest you to report and create this issue on the Google Issue Tracker as need to investigate. Please provide your project number and instance name over there. Please follow this URL that make you able to create a file as private in Google Issue Tracker.

Milad Tabrizi
  • 480
  • 3
  • 15
  • It's almost impossible to find a link to report such an issue in the Console. You should definitely consider adding it for those users which don't have any support plan. For me, the only way to ask about help regarding this issue was to post on StackOverflow. – Vit D Oct 23 '18 at 12:51
  • You can access the link above with your Google/gmail account, and you do not need a support package as issue tracker is meant for reporting defects and feature requests as per this [document](https://cloud.google.com/support/docs/issue-trackers). However, the component in the link is private so you can share your project information. – Milad Tabrizi Oct 24 '18 at 00:34