2

I'm trying to create a flow which as different experiments doing train test split, train, validate, get the best model(among 8 diff algos) and predict. The issue is I need to create a dependency of experiments and I need help in that. I'm aware of azure ml pipelines, but I'm looking for something where we can create a pipeline of pipelines, or something which will help me create a pipeline of multiple experiments(with dependency).

eg for sample pipeline: (train-test-split)->(train[custom,Many-model])->(validate)->(getbest alogos)->(predict) there will be other experiments in between for tasks like registering modls.downloading pickles etc.

akhil dasari
  • 31
  • 2
  • 1
    azure has notion of a pipeline template which can be parameterized and called from either another pipeline or even another template – Scott Stensland Dec 31 '21 at 16:13
  • OP you need to remove your azure-pipelines tag... People are thinking your question is about ADO Pipelines. – David Wu Apr 14 '22 at 13:50

1 Answers1

1

You can achieve using the Azure Pipeline Templates.

Top tip for anyone who tries this: you can define variables at the top level (i.e., in the calling pipeline) and they will be available within the template files. The template files are simply stapled into the pipeline just before it's run, much like server-side includes (or #includes for anyone as old as me). You only need to use template parameters if your individual pipelines share a variable name that needs to be set differently for each one (e.g., "service-name"). Be aware that template parameters (${{}}) are evaluated before the pipeline is run (much like #define constants in C) whereas variables ($()) are evaluated at run-time; so if one or more of your pipelines aren't working as expected, you might need to turn a variable into a template parameter.

Ram
  • 2,459
  • 1
  • 7
  • 14