0

Trying to boot up a VM (notebook) with a GPU (any GPU type) in Google Cloud, I get this error:

"tensorflow-1-15-20210102-230326: Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally. (if this notebook instance needs to use a resource outside of project recsys-296821, in the project that contains the resource, please grant the missed permission to the Cloud AI Platform Notebooks Service Account: service-64420327861@gcp-sa-notebooks.iam.gserviceaccount.com)".

I already tried to increase my total gpu quotas, having already upgraded my account from the Free tier (like in GCP error: Quota 'GPUS_ALL_REGIONS' exceeded. Limit: 0.0 globally and GPU Quota Error even when i have Quota for Nvidia P100 both for region Us-west1 and Europe-west4 from 0 to 1 etc), but the GPU_ALL_REGIONS is grayed out and I can't change it, while still being set to 1, so in theory I should be able to use 1 gpu, but I cannot.

I really don't know how to solve this issue and couldn't find an up to date answer. Thanks in advance.

Filippo Fedeli
  • 25
  • 6
  • 1
    Upgrading your account from Free Tier does not mean your GPU quota has been increased. You must ask for a quota increase. https://cloud.google.com/compute/quotas – John Hanley Jan 02 '21 at 22:34
  • ok, thank you, but I can't seem to be able to ask for a quota increase for all regions. Could you please tell me how to do it precisely? Following that guide I was apparently not able to, as for 'all regions' the option seemed grayed out. Thank you!!! – Filippo Fedeli Jan 02 '21 at 22:38
  • 1
    You request quota increases for a specific region, not globally. If your account is new, only ask for one region and one GPU. – John Hanley Jan 02 '21 at 22:42
  • ok, thanks. Looking at quotas in the regions, I see in theory I already have values different from zero for the regions, but then I cannot use them. I still tried to move one region from 1 to 2 as an experiment, but my request was promptly rejected. What have I done wrong? – Filippo Fedeli Jan 02 '21 at 22:52
  • 1
    GPUs are expensive. Is your account new? If yes, contact Google Cloud Billing to increase your credit (reduce restrictions). If you have GPU quota of 1 in a specific region then launch your services in that region. If you have a problem, create a detailed question showing your work, the problem, errors, etc. For quota related problems you will need to contact Google Cloud Support. – John Hanley Jan 02 '21 at 22:57
  • When did you sent the quota upgrade request? It will take 24hrs to 48 hrs to [activate](https://cloud.google.com/network-connectivity/docs/interconnect/quotas#requesting-additional-quota) the resources for the specific project. – Mahboob Jan 02 '21 at 23:53
  • Thank you, my account is ~ 1 year old and my current project is 1 month old. In the regions, where I supposedly have quota, I can't launch the gpus, as it says my global limit is 0, which is not in the quotas page. The quota upgrade request was instarejected, but I will try to contact the support to try and understand what to do. – Filippo Fedeli Jan 03 '21 at 11:20

1 Answers1

0

According to this GCP official doc, I understand that you might have enough quota to use the resources but it is possible that there might not be available resources in that region. For example, you might still have quota to create VM instance with GPU in us-central1, but there might not be available GPU in that region.

Best possible solutions are:

1- This is a temporary situation that can change anytime based on fluctuating demand, try your request again later.

2- Try to create resources in another zone in the region or in another region.

3- Try to change the shape of the VM you are requesting. For example, you need NVidia P100 at this time, if it works you can use another Nvidia T4

4- Use Compute Engine reservations to reserve the resources within a zone to ensure that the resources you need are available when you need them.

Mahboob
  • 1,877
  • 5
  • 19
  • Thank you for your detailed answer. About number 1-2, the error message indeed changes varying on the zones, as sometimes there truly is no availability. I have tried changing gpus, requesting k80, p100, t4 etc, but nothing changed. About number 4, thank you for pointing it out, but I guess there still is some problem before delving into that. – Filippo Fedeli Jan 03 '21 at 11:21
  • As a last resort you can contact [Sales Rep.](https://cloud.google.com/contact) mentioning your quota issues, OR Call the Google Cloud sales team at: 844-613-7589 Representatives are available Monday—Friday, 8:00 AM - 8:00 PM EST – Mahboob Jan 04 '21 at 03:53