5

It looks like Kubeflow has deprecated all of their TFX components. I currently have some custom Kubeflow components that help launch some of my data pipelines and I was hoping I could use some TFX components in the same kubeflow pipeline. Is there a recommended approach to mix Kubeflow and Tfx components together?

I saw an older PR from Kubeflow deprecating their TFX components: https://github.com/kubeflow/pipelines/issues/3853

It states:

These components were created to allow the users to use TFX components in their KFP pipelines, to be able to mix KFP and TFX components. If your pipeline uses only TFX components, please use the official TFX SDK.

But I actually do need to mix KFP and TFX components, is there a way to do this?

sleepyowl
  • 168
  • 5

1 Answers1

3

The simple answer is no, the long answer is you could, if you hack it. The experience wouldn't be great though.

When you look at an example TFX pipeline, it has it's own Python DSL. As a user, you define the pipeline components the way you want it to run, and at the very end you can change the target runner (Airflow, Beam, and KFP). TFX will compile it's intermediate representation before submitting that to the runner of your choice.

The question then is how can you mix that with other tools. TFX compiles an Argo workflow DAG, similar to if you use the KFP SDK or Couler. When you use the KubeflowDAG runner, you can find the output Argo YAML for the pipeline. If you repeat the same compilation process with your KFP native pipeline, you'll have two Argo YAMLs you can merge together for the specific workload you want.

If you are using MLMD, you may need to do some input/output manipulation to make it all work.

Glad Born
  • 43
  • 5
regularlearner
  • 419
  • 3
  • 11