15

I was exploring kubeflow pipelines and Vertex AI pipelines. From what I understand, Vertex AI pipelines is a managed version of kubeflow pipelines so one doesn't need to deploy a full fledged kubeflow instance. In that respect, pricing aside, Vertex AI pipelines is a better choice. But then, in kubeflow, one can create experiments, an equivalent for which I have not found in Vertex AI pipelines. The only kubeflow features that Vertex AI does not support that I have been able to spot in the documentation are "Cache expiration" and "Recursion" but they do not mention anything about experiments. Makes me wonder if there are other differences that are worth considering when deciding between the two.

racerX
  • 930
  • 9
  • 25

3 Answers3

10

The team I work with has been investigating Vertex AI and comparing with KubeFlow for the past few months. As you pointed out, Vertex AI experiments are not the same as KubeFlow's. Vertex's experiments are just an interface for Tensorboard instances and Vizier hyperparameter tuning.

There seems to be no equivalent in Vertex AI for grouping pipeline runs into experiments. However, as even the authors of KubeFlow for Machine Learning point out, KubeFlow's own experiment tracking features are pretty limited, which is why they favor using KubeFlow alongside MLflow instead.

Some other differences I have noticed:

  • Low level access: Vertex, being a managed service, does not expose the instances where the pipeline steps run, whereas in KubeFlow they are plain old Pods inside of Kubernetes, so in theory you should have more low level control.
  • Pipeline registry/pipeline versions: in KubeFlow, you can store pipelines and track versions of compiled pipelines within the tool itself, along with descriptions for the changes. In Vertex, there is no equivalent and you would have to store pipeline objects in GCS (but I would argue you would be better off tracking pipeline code in a version control system anyways).
  • Scheduled/recurring pipeline runs are not yet supported on Vertex.

Other than these and what you already mentioned, Vertex seems to be at feature parity with KubeFlow in pipeline execution features, with the great advantage of not having to manage a Kubernetes cluster.

Jota Porras
  • 163
  • 5
  • 1
    I am pretty sure I had created a scheduled/recurring Vertex AI pipeline before, see this https://stackoverflow.com/questions/68793294/how-to-schedule-repeated-runs-of-a-custom-training-job-in-vertex-ai, which basically used Cloud Scheduler under the hood. But when I check the docs now, there is no reference to this, maybe they removed this support. – racerX Dec 16 '21 at 14:30
  • how was your experience w.r.t. speed. Pipelines is supposed to be useful for experimentation, trying out different models, strategies etc. Vertex AI pipelines for a simple toy flow took me anywhere from 2 mins to ~10 mins, much slower than say, running something locally. Have not been able to experiment with kubeflow yet so don't know if it's any faster (the local standalone deployment or a full fledged kubernetes deployment). – racerX Dec 16 '21 at 14:37
  • 1
    KubeFlow pipeline stages take a lot less to set up than Vertex in my experience (seconds vs couple of minutes). This was expected, as stages are just containers in KF, and it seems in Vertex full-fledged instances are provisioned to run the containers. For production scenarios it's negligible, but for small experiments definitely not worth. One alternative I'm considering is to test pipelines locally using KF on minikube and then VAI for larger stuff. Either that or just test the containers separately on your local docker. – Jota Porras Dec 16 '21 at 16:36
2

Actually, Vertex AI is a serverless platform to execute pipelines built in:

Kubeflow Pipelines SDK v1.8.9 or higher, or TensorFlow Extended v0.30.0 or higher.

In fact, almost any Kubeflow pipeline you are used to create will run pretty well in Vertex AI, You just have to take into consideration the aspects you already mentioned in the docs (io, dsl and storge) (which kind of common sense, since you can't from disk and have to use an external file source).

So, basically is just where you run it. The development does not change in fact if you use kubeflow (i.e import kfp ).

UPDATE: Experiments are supported also as almost all features. Just check here at: Experiments – Vertex AI – Google Cloud Platform

enter image description here

some references:

Building a pipeline  |  Vertex AI  |  Google Cloud

Running an Experiment | Kubeflow

Experiments – Vertex AI – Google Cloud Platform

ewertonvsilva
  • 1,795
  • 1
  • 5
  • 15
  • I understand that, but what I'd like to know is, based on everyone's experience, what features might be missing in Vertex AI pipelines, like in kubeflow, one can organize pipeline runs into logical groups. Nothing about this is mentioned in Vertex AI. – racerX Dec 15 '21 at 22:09
  • Experiments are also supported. Please check the **updade** on the answer @racerX. – ewertonvsilva Dec 16 '21 at 10:35
  • Not the same, the experiments in kubeflow allow you to group your pipelines, i.e., when you run a pipeline in kubeflow, you can associate that run with a specific experiment, did not find a way to do the same in Vertex. Also, Vertex experiments is specifically for use with tensorboard – racerX Dec 16 '21 at 12:47
0

From my understanding, the feature you are mentioning should be obtained in VertexAI with:

  • tensorboard.

So at first Tensorboard is very basic so it needs to be customized with custom variables (like learning rate) and other custom graphs.

Although, the very first step is obviously to enable it. As it won't be seen or be greyed out using Iris classification example.

Alsushi
  • 373
  • 3
  • 14