6

I'm developing a complex pipeline in Vertex AI using Pipelines and components. I would like to import some custom modules and functions I developed for this use case. Unfortunately, I cannot figure out how to import those custom functions in my code without creating ad-hoc Docker images or without publishing my code on public repositories like PyPi.

There are two pain points in pasting those custom functions' code in each component:

  1. The code becomes huge and difficult read
  2. The function's code completely loses the maintenability because at each small change, I have to replicate it for each component.
Andrea Cola
  • 127
  • 1
  • 8
  • 1
    There are two ways to use Custom modules, one is using a [custom container](https://cloud.google.com/vertex-ai/docs/training/create-custom-container) and the other one is uploading it into a public repository and install the module into vertex. – Jose Gutierrez Paliza Aug 16 '22 at 16:36
  • 1
    @JoseGutierrezPaliza both ways are not very fast and comfortable during the development phase. I agree that when the code is ready and tested, the smartest way is to containerize the component. But, during the development phase is very boring and time-wasting. – Andrea Cola Aug 17 '22 at 19:09

1 Answers1

4

KFP supports Function-based components which can run a standalone function in a generic python image.

Michael Hu
  • 96
  • 3
  • Yes, I've already implemented by pipeline using function-based components but the main problem is the size of these components. In each component, I have to include a lot of custom support functions, and my question is: is there a method to systematically import that functions in each component without creating a PyPi repository? – Andrea Cola Aug 24 '22 at 10:24
  • 2
    I see why function-based components would be unappealing. Wrapping all your code in a single image and uploading it to Artifact Registry would be the most straightforward solution in that case. You would then create component specs with different python commands to execute each function, all pointed at the same image. – Michael Hu Aug 25 '22 at 04:08