Azure ML workspace: how to publish pipeline to existing endpoint instead of creating new

Question

I'm working on deploying an inference pipeline in Azure machine learning workspace.

I have created a pipeline using a couple of PythonScriptSteps and want to automate the pipeline publishing using CI/CD.

Reference: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-pipelines#publish-a-pipeline

pipeline = Pipeline(workspace=workspace, steps=[step1, step2])
pipeline_endpoint = pipeline.publish(name='deployment-test', version=1)

Every time I publish, it is creating new endpoints but I want to deploy to existing so that nothing has to be changed in the consumer end.

The Azure documentation of the versioned pipeline endpoint is not correct currently. For more information, see this [GitHub issue](https://github.com/MicrosoftDocs/azure-docs/issues/59931). — tuomastik, May 19 '21 at 08:07

score 4 · Answer 1 · edited May 18 '21 at 13:43

4

PipelineEndpoint can be used to update a published pipeline while maintaining the same endpoint. PipelineEndpoint provides a way to keep track of PublishedPipelines using versions. PipelineEndpoint uses endpoint with version information to trigger an underlying published pipeline. Pipeline endpoints are uniquely named within a workspace.

edited May 18 '21 at 13:43

tuomastik

4,559
5
36
48

answered Jan 31 '21 at 02:09

Ram

2,459
1
7
14

2

Problem with PipelineEndpoint is that I cannot get the run history as run history is tied to a pipeline. Is it possible to get run history from PipeLineEndpoint? Additionally, Azure ML cli does not support PipelineEndpoint which makes it harder to deploy new pipelines using CI/CD. – Luniam Feb 05 '21 at 11:39

score 3 · Answer 2 · answered Jun 03 '21 at 11:53

I had the same problem as you and managed to get it to work by doing the following:

First initialize the endpoint:

pipeline = Pipeline(workspace=ws, steps=steps)
published = pipeline.publish(
            name="name"
        )
pipeline_endpoint = PipelineEndpoint.publish(
            workspace=ws,
            name="My endpoint name",
            pipeline=published,
            description="Endpoint to my pipeline",
       )
pipeline_endpoint.add_default(published)

The next time you run this you instead run:

pipeline = Pipeline(workspace=ws, steps=steps)
published = pipeline.publish(
            name="name"
)
pipeline_endpoint = PipelineEndpoint.get(
     workspace=ws, name="My endpoint name"
)
pipeline_endpoint.add_default(published)

add_default makes sure that the latest version of the pipeline is used in the endpoint.

But there's no way to republish a PipelineEndpoint without first publishing the Pipeline run? My use case is that I'd like to republish a pipeline definition (the Pipeline() object) without having to publish a run of the pipeline before publishing it as a PipelineEndpoint — geominded, Mar 29 '22 at 02:05
For example, you can submit `PipelineEndpoint.publish(workspace, pipeline=Pipeline(workspace, steps=[step1, step2]), name='pipeline_name')`, but you can't later add a new pipeline definition using `PipelineEndpoint.add_default(Pipeline(workspace, steps=[step1, step2]))` .add_default ONLY accepts PublishedPipeline objects — geominded, Mar 29 '22 at 02:09

Azure ML workspace: how to publish pipeline to existing endpoint instead of creating new

2 Answers2