Tensorflow: How to add a property in execution object in MLMD MetadataStore?

Question

I'm using the MLMD MetadataStore to manage the data pipelines and I need to add an execution property in MLMD to get this property later.

I'm trying add with this:

from ml_metadata.proto import metadata_store_pb2
from ml_metadata.metadata_store import metadata_store

def create_mlmd_database_in_memory():
    connection_config = metadata_store_pb2.ConnectionConfig()
    connection_config.fake_database.SetInParent()  # Sets an empty fake database proto.
    store = metadata_store.MetadataStore(connection_config)
    return store

store = create_mlmd_database_in_memory()

# Create an ExecutionType, e.g., Trainer
trainer_type = metadata_store_pb2.ExecutionType()
trainer_type.name = "TrainerTest"
trainer_type_id = store.put_execution_type(trainer_type)

# Register the Execution of a Trainer run
trainer_run = metadata_store_pb2.Execution()
trainer_run.type_id = trainer_type_id
trainer_run.properties["pipeline_name"].string_value = "PIPELINE_NAME"
[run_id] = store.put_executions([trainer_run])

But I'm getting this error:

InvalidArgumentError: Found unknown property: pipeline_name

Does anyone know how to do this?

score 0 · Answer 1 · answered Mar 31 '23 at 02:22

Is missing add the property in execution type too.

Like this

# Create an ExecutionType, e.g., Trainer
trainer_type = metadata_store_pb2.ExecutionType()
trainer_type.name = "TrainerTest"
trainer_type.properties["pipeline_name"] = metadata_store_pb2.STRING
trainer_type_id = store.put_execution_type(trainer_type)

# Register the Execution of a Trainer run
trainer_run = metadata_store_pb2.Execution()
trainer_run.type_id = trainer_type_id
trainer_run.properties["pipeline_name"].string_value = "PIPELINE_NAME"
[run_id] = store.put_executions([trainer_run])

Reference https://www.tensorflow.org/tfx/guide/mlmd:

An Execution is a record of a component run or a step in an ML workflow and the runtime parameters. An execution can be thought of as an instance of an ExecutionType. Executions are recorded when you run an ML pipeline or step.

Tensorflow: How to add a property in execution object in MLMD MetadataStore?

1 Answers1