How can I programmatically create a Kubeflow recurring run from a pipeline function?

Question

I am trying to create a recurring kubeflow pipeline run as follows:

from kfp import compiler
compiler.Compiler().compile(
    pipeline_func=my_pipeline,
    package_path='pipelines/my_pipeline.tgz')

from kfp.v2.google.client import AIPlatformClient
api_client = AIPlatformClient(project_id='...',
                             region='...')

api_client.create_schedule_from_job_spec(
    job_spec_path='pipelines/my_pipeline.tgz',
    schedule='* * * * *',
    time_zone='UTC',
    parameter_values=arguments
)

The first command creates the pipeline spec in YAML, but the second one expects JSON.

How otherwise can I create the recurring run programmatically [rather than via the UI]?

score 2 · Accepted Answer · answered Nov 02 '21 at 13:19

You're using the v1 kfp.compiler which indeed produces a yaml output and your API client is v2, which indeed expects a json input. Use the kfp.v2.compiler to be consistent with the versions, the later saves the pipeline in json, as expected by your api_client.

You can find an example of creating a v2 pipeline on gcp here.

How can I programmatically create a Kubeflow recurring run from a pipeline function?

1 Answers1