matching engine index creation failed

Question

I want to create Matching Engine Index in Vertex AI, I was following this article step by step and got stuck in matching engine index creation

https://medium.com/google-cloud/q-a-with-your-docs-a-gentle-introduction-to-matching-engine-palm-bbbb6b0cff7b

It just says creation failed, hovering over the icon doesn't give any information.

Here is the generated embeddings file according to the article

• I have stored data.json in the GCS bucket with the required storage.
• I have made sure my Google project and storage are under the same regions - us-central1
• here is my index_metadata.json file

Here is the command I am using to create an index

gcloud ai indexes create --metadata-file=./index_metadata.json --display-name=my-index-5 --project=my-project --region=us-central1

Don't know what is causing failure and where to find the information

Answer 1

I faced a similar issue (as per my original comment to the post) and it seems the remedy for my problem was my ".json" (emphasis on the quote marks).

It would appear the expected data to index needs to be in a ".json-like" format and not a true .json format. Examples:

Good proper data:

{"id":"0","embedding":[ 0.1,0.1,0.1 ...]}
{"id":"1","embedding":[ 0.1,0.1,0.1 ...]}
{"id":"2","embedding":[ 0.1,0.1,0.1 ...]}

"Bad"/invalid data:

{"id":"0","embedding":[ 0.1,0.1,0.1 ...]}**,**
{"id":"1","embedding":[ 0.1,0.1,0.1 ...]}**,**
{"id":"2","embedding":[ 0.1,0.1,0.1 ...]}**,**

Notice the commas separating the lines of data... I couldn't find any record of this requirement, but it seems modifying my data to fit this solved it, at least for me. You didn't provide a full copy of your data to see, so I cannot know for sure if this is the issue you face.

I hope this insight helps you out!

Answer 2

Without Logging data is very difficult to know what is going on.

This notebook will help you, it presents a good walkthrough on how to deploy a Matching Engine Index:

Matching Engine

There are several points to look for:

• the network
• if you are calling from us-central1
• peering
• json format
• embeddings dimension
• pb2 files
• protobuf
• grPC

Answer 3

Have the same issue when creating index with Nodejs library: @google-cloud/aiplatform

The below sample has been taken from official nodejs sample reference:

    const parent = `projects/${projectId}/locations/${location}`;
    
    let index = {
      displayName: "sdfasfas",
      metadata: {
        contentsDeltaUri: "gs://redacted/vectors",
        config: {
          **dimensions: 768,**
          approximateNeighborsCount: 150,
          distanceMeasureType: "COSINE_DISTANCE",
          shardSize: "e2-standard-2",
          algorithm_config: {
            treeAhConfig: {
              leafNodeEmbeddingCount: 5000,
              leafNodesToSearchPercent: 3,
            },
          },
        },
      },
    };

    console.log("index =>", index);

    let request = {
      parent,
      index,
    };

    console.log("request =>", request);```
    
    // Get and print out a list of all the endpoints for this resource
    const [result] = await client.createIndex(request);'

The above code errors with below message:

    {
  code: 9,
  details: 'dimensions is required but missing from Index metadata.', 
  metadata: Metadata {
    internalRepr: Map(2) {
      'endpoint-load-metrics-bin' => [Array],
      'grpc-server-stats-bin' => [Array]
    },
    options: {}
  }
}

'dimensions' key and value are a part of the JSON object. Not sure why the error messages still says "'dimensions is required but missing from Index metadata.".