0

I'm having issues starting / creating new user-managed notebooks in Vertex AI > Workbench, where I'll end up with this error and can't even open JupyterLabs:

[Vertex AI] - Restarting notebook NAME: ZONE does not have enough resources available to fulfill the request. Retry later or try another zone in your configurations.

I didn't have this problem last month when I was still under the free trial. I also noticed this error only come up when I try to install the GPU in the notebook.

So far, I've done the following:

  • Change payment method (got an email that asked me to update payment settings at the end of the trial last month)
  • Create new notebook in different zones and cluster over the past week, all give the same error when I attach a GPU.
  • In Quotas, changed 'GPU (all regions)' from 1 to 0, and then back to 1.
  • In Quotas, changed 'VM instances (default)' from 24 to 48, and then back to 24.
  • Created several new projects, didn't work either.

I also created a snapshot of both the data and boot of the notebook, but don't know how to use it to recreate it in Vertex AI again.

Have you faced a similar issue before, how did you fix it? Thanks

Wilson Liu
  • 29
  • 3
  • Quota in Vertex User-managed Notebooks depends on Compute Engine directly. What zone, machine type and what GPU type are you using? Please check the specific region quota. – gogasca Jan 14 '22 at 00:46
  • Hi there, I figured out what was going on. Basically, I couldn't restart the notebook with the GPU because there's a GPU shortage in that zone. The solution (for me) was either wait till after midnight and start then, or create a new notebook in a region on the other side of the world, where it's near / after midnight around there. Not the best solution, but at least you can use the GPU for your work again... Hope it helps. – Wilson Liu Jan 26 '22 at 00:18

1 Answers1

-1

As mentioned in the comments the asker discovered the root cause was a GPU shortage in the region

I couldn't restart the notebook with the GPU because there's a GPU shortage in that zone. The solution (for me) was either wait till after midnight and start then, or create a new notebook in a region on the other side of the world, where it's near / after midnight around there. Not the best solution, but at least you can use the GPU for your work again...

Dennis Jaheruddin
  • 21,208
  • 8
  • 66
  • 122