0

We’re using a managed Kubeflow Pipelines (KFP) instance, created with GCP’s AI Platform Pipelines, with a managed MySQL instance, created with Cloud SQL. We’re also using Cloud Build to build and run our KFP pipelines. We’d like to add a Cloud Build step that, after a KFP pipeline run finishes, runs a script that uses the MLMD API to query for that pipeline run’s metadata. We have an MLMD script that can successfully query for metadata when run manually on a GCP VM. The issue has been getting that script to run in Cloud Build.

The first approach was to create the mlmd.metadata_store.MetadataStore object using MySQL connection information, e.g.,

connection_config = metadata_store_pb2.ConnectionConfig()
connection_config.mysql.host = [IP address]
connection_config.mysql.port = 3306
connection_config.mysql.database = "kubeflow_experiments_metadata_metadata"
connection_config.mysql.user = [user]
connection_config.mysql.password = [password]
store = mlmd.metadata_store.MetadataStore(connection_config)

This runs fine on a VM. It appears, however, that the Cloud SQL Proxy is required for this to work in Cloud Build. Using this Cloud Functions codelab as an example, I was able to run a script in Cloud Build that used sqlalchemy to connect to MySQL via the proxy. The method for connecting sqlalchemy with the proxy, however, seems to be incompatible with the above MLMD API. It looks something like this:

driver_name = "mysql+pymysql"
query_string = dict({"unix_socket": "/cloudsql/{}".format(connection_name)})
db = sqlalchemy.create_engine(
    sqlalchemy.engine.url.URL(drivername=driver_name, username=[user], password=[password], database="kubeflow_experiments_metadata_metadata", query=query_string),
    pool_size=5,
    max_overflow=2,
    pool_timeout=30,
    pool_recycle=1800,
)

The second approach uses the MLMD gRPC service that’s deployed with KFP. First, I port forward the service:

kubectl port-forward svc/metadata-grpc-service 8080:8080

then create the mlmd.metadata_store.MetadataStore using the MLMD gRPC API:

connection_config = metadata_store_pb2.MetadataStoreClientConfig(
    host="127.0.0.1",
    port=8080,
)
store = mlmd.metadata_store.MetadataStore(connection_config)

Again, this works fine on a VM. I’m not sure, however, how to connect to the gRPC service from Cloud Build. My experience with gRPC and Kubernetes is limited, so I wouldn’t be surprised if there was a straightforward solution.

Any advice would be greatly appreciated!

hahns0lo
  • 1
  • 2
  • Can you expand on this statement?: "The method for connecting sqlalchemy with the proxy, however, seems to be incompatible with the above MLMD API." – kurtisvg Feb 18 '21 at 01:50
  • Sure. The Cloud Functions codelab I referenced above appears to configure the Cloud SQL Proxy to listen on a Unix port in the `/cloudsql` directory. This socket is how the sqlalchemy code connects to MySQL via the proxy. The MLMD API for connecting to MySQL, however, only appears to accept an IP address and port. Hope that helps a bit. – hahns0lo Feb 19 '21 at 01:46
  • You might be interested to know that the Cloud SQL proxy supports listening on a TCP socket as well: https://cloud.google.com/sql/docs/mysql/connect-admin-proxy#tcp-sockets For Cloud Build you can use something like this to run the proxy: https://stackoverflow.com/a/58441728/9414803 – kurtisvg Feb 19 '21 at 17:24
  • Cool, thanks for the tip! To run the sqlalchemy script in Cloud Build, I used [this image](https://github.com/GoogleCloudPlatform/ruby-docker/tree/master/app-engine-exec-wrapper) that installs and configures the Cloud SQL Proxy. [This line](https://github.com/GoogleCloudPlatform/ruby-docker/blob/553cd02/app-engine-exec-wrapper/execute.sh#L89) runs the proxy with a Unix socket, so looks like this image can't be used with a TCP socket. The gRPC solutions works, for now, but I'll probably return to this and take a look at the TCP socket approach you pointed to. Thanks, again! – hahns0lo Feb 20 '21 at 16:16

1 Answers1

0

Through other avenues, I was pointed to this article, which contains an example of how to port-forward KFP's ml-pipeline service in Cloud Build. I had to make one minor modification, which was to remove the "-n" and "kubeflow" arguments to the kubectl port-forward command. This specifies to kubectl to use the "kubeflow" namespace. GCP's AI Platform Pipelines, however, appears to create a "default" namespace when it deploys your KFP instance.

hahns0lo
  • 1
  • 2