Stack Overflow
Stack Overflow

Questions tagged [kubeflow-pipelines]

Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers.

273 questions
23
votes
9 answers

Sudden ImportError: cannot import name 'appengine' from 'requests.packages.urllib3.contrib error on pipeline

My pipelines and schedulers were running smoothly without any problems. After I went out to lunch, I changed the number of epochs a Neural Network would run, save the .yaml file again and leave it in the bucket named "budgetff". Afterwards,…
filipe
  • 275
  • 1
  • 1
  • 8
18
votes
7 answers

Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled

While running kubeflow pipeline having code that uses tensorflow 2.0. below error is displayed at end of each epoch W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled:…
Radhi
  • 6,289
  • 15
  • 47
  • 68
15
votes
3 answers

Kubeflow vs Vertex AI Pipelines

I was exploring kubeflow pipelines and Vertex AI pipelines. From what I understand, Vertex AI pipelines is a managed version of kubeflow pipelines so one doesn't need to deploy a full fledged kubeflow instance. In that respect, pricing aside, Vertex…
racerX
  • 930
  • 9
  • 25
13
votes
2 answers

What are the differences between airflow and Kubeflow pipeline?

Machine learning platform is one of the buzzwords in business, in order to boost develop ML or Deep learning. There are a common part workflow orchestrator or workflow scheduler that help users build DAG, schedule and track experiments, jobs, and…
Kevin Su
  • 542
  • 2
  • 6
  • 24
10
votes
1 answer

Vertex AI Model Batch prediction, issue with referencing existing model and input file on Cloud Storage

I'm struggling to correctly set Vertex AI pipeline which does the following: read data from API and store to GCS and as as input for batch prediction. get an existing model (Video classification on Vertex AI) create Batch prediction job with input…
zdenulo
  • 346
  • 3
  • 14
10
votes
3 answers

How to pass data or files between Kubeflow containerized components in python

I'm exploring Kubeflow as an option to deploy and connect various components of a typical ML pipeline. I'm using docker containers as Kubeflow components and so far I've been unable to successfully use ContainerOp.file_outputs object to pass results…
Ash
  • 969
  • 3
  • 16
  • 28
8
votes
6 answers

How to get the id of the run from within a component?

I'm doing some experimentation with Kubeflow Pipelines and I'm interested in retrieving the run id to save along with some metadata about the pipeline execution. Is there any way I can do so from a component like a ContainerOp?
DSF
  • 83
  • 1
  • 3
7
votes
3 answers

kubeflow pipeline dynamic output list as input parameter

I use a ParallelFor over a dynamic list. I want to collect all the outputs from the loop, and pass them to another ContainerOp. Something like the following, which obviously does not work, since the outputs list is will be static. with…
user3599803
  • 6,435
  • 17
  • 69
  • 130
5
votes
1 answer

Is it possible to mix kubeflow components with tensorflow extended components?

It looks like Kubeflow has deprecated all of their TFX components. I currently have some custom Kubeflow components that help launch some of my data pipelines and I was hoping I could use some TFX components in the same kubeflow pipeline. Is there a…
sleepyowl
  • 168
  • 5
5
votes
2 answers

Sharing secrets in Kubeflow pipeline

I want to share some secrets with my Kubeflow pipeline so I can use them as environment variables in my containers. I've written a pipeline-secrets.yaml that looks like this: apiVersion: v1 kind: Secret metadata: name: pipeline-secrets …
João Areias
  • 1,192
  • 11
  • 41
5
votes
1 answer

How do I define pipeline-level volumes in kubeflow pipelines to share across components?

The kubernetes Communicating between containers tutorial defines the following pipeline yaml: apiVersion: v1 kind: Pod metadata: name: two-containers spec: restartPolicy: Never volumes: <--- This is what I need - name:…
RunOrVeith
  • 4,487
  • 4
  • 32
  • 50
5
votes
2 answers

Aggregate results when using Kubeflow Pipelines kfp.ParallelFor

What is a good pattern for aggregating the results from Kubeflow Pipleine kfp.ParallelFor?
Jet Basrawi
  • 3,185
  • 2
  • 15
  • 14
4
votes
0 answers

Convert/wrap Spacy model into a Tensorflow model

Is it possible to convert a saved spacy model into a TensorFlow model? Spacy provides a wrapper to work with TensorFlow models. Mentioned here. Can this be done the other way around? I need to convert a Spacy model into a TensorFlow model or wrap it…
Shashank Yadav
  • 174
  • 1
  • 11
4
votes
0 answers

download from URL Kubeflow pipeline

I have been trying to create a kubeflow pipeline in my local machine and created a simple one component pipeline to download data from a given url. import kfp import kfp.components as comp downloader_op =…
user395882
  • 665
  • 2
  • 7
  • 16
4
votes
2 answers

Reading Data in Vertex AI Pipelines

This is my first time using Google's Vertex AI Pipelines. I checked this codelab as well as this post and this post, on top of some links derived from the official documentation. I decided to put all that knowledge to work, in some toy example: I…
David Espinosa
  • 760
  • 7
  • 21
1
2 3
…
18 19