0

I have a kubeflow pipeline which trains custom (i.e. not based on sklearn / tensorflow etc. classes) ml model. Now I would like to add serving at the end of the pipeline. I.e. I want to have a service in my Kubernetes cluster which uses the model to answer prediction requests and this service should be updated with a new model after each pipeline run.

As far as I know to serve a custom model I should:

  1. Wrap my model into kfserving.KFModel class

  2. Create docker image with the wrapper from 1) running

  3. Create InferenceService endpoint with image from 2)

Is there any cloud agnostic way to do this in a Kubeflow component? (so basically the component must be able to build docker images)

Is there some better way to achieve my purpose?

Maybe I should move steps 1-3 outside of pipeline component and just create a component which would trigger external execution of 1-3. Can this be done?

CoolBeans
  • 20,654
  • 10
  • 86
  • 101
Tomasz Cakala
  • 1
  • 1

2 Answers2

0

I can't speak to Kubeflow in particular, but https://buildpacks.io/ provides a general-purpose way to build containers that satisfy certain input criteria (for example, "is a python program with a main and a requirements.txt"). It's also possible (but more complicated) to create a new buildpack (for example, to take "python code that implements kfserving.KFModel and wrap a main and whatever else is needed around it). I've done this a few times for python for demos/etc:

https://github.com/evankanderson/klr-buildpack https://github.com/evankanderson/pyfun

Note that these aren't production-grade, just me playing around for a day or three.

You can build buildpacks locally with the pack command, or on a cluster using several technologies. There's detailed documentation for 5 build options here: https://buildpacks.io/docs/tools/, along with a longer list of "supported platforms" at the bottom of https://buildpacks.io/features/.

E. Anderson
  • 3,405
  • 1
  • 16
  • 19
  • the problem is it still needs docker - so I can't use it inside Kubeflow component. I can automate everything I need with a bash script. Alas, I'd like to do it inside Kubeflow component. – Tomasz Cakala Nov 02 '21 at 16:47
  • I'll admit I'm not familiar with the component, but buildpacks can be used and built without access to the Docker socket; this is how the Tekton buildpack task works, for example. – E. Anderson Nov 03 '21 at 01:53
0

You should definitely move steps 1-3 outside of the Kubeflow Pipeline, building the docker images for your custom model server shouldn't be done on every pipeline run.

Having said that, your custom image should load the blessed model of a run from an external source, ie S3/GS/minio bucket.

Are you sure that your model is built on a framework that is not in the list of model servers that Kserve supports already and you need to create a custom model server?

Theofilos Papapanagiotou
  • 5,133
  • 1
  • 18
  • 24