Stack Overflow
Stack Overflow

Questions tagged [kfp]

Questions related to the kubeflow pipelines python SDK

The kubeflow pipelines python SDK kfp is used to programmatically interact with kubeflow pipelines and the underlying kubernetes cluster.

34 questions
8
votes
1 answer

Kubeflow Pipeline Termination Notificaiton

I tried to add a logic that will send slack notification when the pipeline terminated due to some error. I tried to implement this with ExitHandler. But, seems the ExitHandler can’t dependent on any op. Do you have any good idea?
Wenmin Wu
  • 1,808
  • 12
  • 24
5
votes
1 answer

How do I define pipeline-level volumes in kubeflow pipelines to share across components?

The kubernetes Communicating between containers tutorial defines the following pipeline yaml: apiVersion: v1 kind: Pod metadata: name: two-containers spec: restartPolicy: Never volumes: <--- This is what I need - name:…
RunOrVeith
  • 4,487
  • 4
  • 32
  • 50
3
votes
1 answer

How can I programmatically create a Kubeflow recurring run from a pipeline function?

I am trying to create a recurring kubeflow pipeline run as follows: from kfp import compiler compiler.Compiler().compile( pipeline_func=my_pipeline, package_path='pipelines/my_pipeline.tgz') from kfp.v2.google.client import…
jtlz2
  • 7,700
  • 9
  • 64
  • 114
2
votes
3 answers

How to test kfp components with pytest

I'm trying to local test a kubeflow component from kfp.v2.ds1 (which works on a pipeline) using pytest, but struggling with the input/output arguments together with fixtures. Here is a code example to illustrate the issue: First, I created a fixture…
Gabriel Caldas
  • 371
  • 2
  • 13
2
votes
1 answer

Set the name for each ParallelFor iteration in KFP v2 on Vertex AI

I am currently using kfp.dsl.ParallelFor to train 300 models. It looks something like this: ... models_to_train_op = get_models() with dsl.ParallelFor(models_to_train_op.outputs["data"], parallelism=100) as item: prepare_data_op =…
Optimus
  • 1,354
  • 1
  • 21
  • 40
2
votes
0 answers

Submitting a pipeline to microk8s.kubeflow, using kfp.Client python api

I installed microk8s and is working. I can upload and run a pipeline using the UI. Now I installed kfp python api (issues exist with the cli too) and I'm trying to submit a pipeline using the cli. Here is the api for the client function: def…
Hamid
  • 59
  • 2
2
votes
1 answer

Kubeflow Multiple images components in pipeline

I want to write a pipeline in Kubeflow pipeline that has 2 components: A and B The output of A is list of image path. I want to run a docker image (B) for each image path From what I see the dsl.ContainerOp of B can wait for output of A, but I don't…
asaf
  • 958
  • 1
  • 16
  • 38
1
vote
1 answer

Cannot install apache-beam==2.35.0, apache-beam[gcp]==2.35.0 apache-beam[gcp]==2.47.0 and apache-beam[gcp]==2.48.0 versions have conflict dependencis

I have a kfp component that installs apache-beam==2.35.0. I run it in Vertex AI on GCP pipelines. Component header: @component( base_image="dockerhub/python:3.7.12", packages_to_install=["tensorflow-data-validation==1.6.0",…
Maha Amin
  • 13
  • 4
1
vote
1 answer

Error when importing sklearn in pipeline component

When I run this simple pipeline (in GCP's Vertex AI Workbench) I get an error: ModuleNotFoundError: No module named 'sklearn' Here is my code: from kfp.v2 import compiler from kfp.v2.dsl import pipeline, component from google.cloud import…
gosia
  • 29
  • 3
1
vote
0 answers

How can I get ImageDatasetImportDataOp to update labels?

In a Vertex AI pipeline I am updating an image dataset, thus: ds_op = gcc_aip.ImageDatasetImportDataOp( project=project, dataset=get_dataset_id_op.outputs['dataset'], gcs_source=DATASET_PATH, …
schoon
  • 2,858
  • 3
  • 46
  • 78
1
vote
1 answer

What is difference between the KFP SDK v2, v2 namespace, and v2 compatible mode?

What is the difference between KFP SDK v2, the v2 namespace in KFP SDK v1, and v2 compatible mode?
Connor McCarthy
  • 41
  • 4
1
vote
1 answer

How do I get a model id in a a Vertex AI pipeline?

Part of my pipeline: training_op = gcc_aip.AutoMLTabularTrainingJobRunOp( project=project, display_name=display_name, optimization_prediction_type="classification", budget_milli_node_hours=1000, …
schoon
  • 2,858
  • 3
  • 46
  • 78
1
vote
0 answers

Why do I get There are no registered serializers for type "google.VertexEndpoint"

I am trying to deploy a model using ModelDeployOp in a Vertex AI pipeline component. My code is: parent = client.common_location_path(project=project, location=location) request = aiplatform_v1.ListEndpointsRequest(parent=parent) page_result =…
schoon
  • 2,858
  • 3
  • 46
  • 78
1
vote
0 answers

How to build a docker from within a kfp component using python docker lib

i have a kfp pipeline with a component from which i want to build an image and push it somewhere to be used in a later component: kfp pipeline: ... --> [Build image comoponent2] --> How can i write a kfp component to do that? i have started some…
Tsvi Sabo
  • 575
  • 2
  • 11
1
vote
1 answer

kfp.v2 compile NotImplementedError

I'm following this tutorial https://cloud.google.com/vertex-ai/docs/pipelines/build-pipeline I'm running this in a Workbench from Vertex IA. When I run this block of code from kfp.v2 import compiler compiler.Compiler().compile(pipeline,…
Godoy32
  • 35
  • 5
1
2 3