1

is it possible to control the budget for each training session in azure ML. Lets say I am training a model and before starting the run, I want to set a budget of X dollars. and if it crosses, then my model training should pause unless I increase the budget to X+Y dollars.

Once the budget is increased, then only model training should resume. Is there any raw rest HTTP API or SDK options to control this?

Onki
  • 1,879
  • 6
  • 38
  • 58

1 Answers1

0

Today, there is no way to control cost at the granularity of a job.

Here is what can be done today:

  1. You can define quotas and usage both at the subscription and the workspace level (i.e. control which compute can be used) -- this is done in https://portal.azure.com enter image description here

  2. You can set up budgets on the resource group level with Cost Management -- but these are just to control and alert and won't limit any jobs (meaning, that a resource group can exceed it's budget). enter image description here

  3. You can set up a spending limit on the subscription level, but that shuts down most/all services that cost money, so not a great option either. https://learn.microsoft.com/en-us/azure/cost-management-billing/manage/spending-limit

Daniel Schneider
  • 1,797
  • 7
  • 20