Update has no effect on scores (Prediction API)

Question

I'm experimenting with the language_id.txt dataset from the Google Prediction example. Right now I'm trying to update the model with the following method:

def update(label, data)
  input = @prediction.trainedmodels.update.request_schema.new
  input.label = label
  input.csv_instance = [data]
  result = @client.execute(
    :api_method => @prediction.trainedmodels.update,
    :parameters => {'id' => MODEL_ID},
    :headers => {'Content-Type' => 'application/json'},
    :body_object => input
  )
  assemble_json_body(result)
end

(This method is based on some Google sample code.)

My problem is that these updates have no effect. Here are the scores for This is a test sentence. regardless of how many updates I run:

{
   "response":{
      "kind":"prediction#output",
      "id":"mymodel",
      "selfLink":"https://www.googleapis.com/prediction/v1.5/trainedmodels/mymodel/predict",
      "outputLabel":"English",
      "outputMulti":[
         {
            "label":"English",
            "score":0.420937
         },
         {
            "label":"French",
            "score":0.273789
         },
         {
            "label":"Spanish",
            "score":0.305274
         }
      ]
   },
   "status":"success"
}

Per the disclaimer at the bottom of "Creating a Sentiment Analysis Model", I have made sure to update at least 100 times before expecting any changes. First, I tried using a single sentence and updating it 1000 times. Second, I tried using ~150 unique sentences drawn from Simple Wikipedia and updated with each once. Each update was "successful":

{"response":{"kind":"prediction#training","id":"mymodel","selfLink":"https://www.googleapis.com/prediction/v1.5/trainedmodels/mymodel"},"status":"success"}

but neither approach changed my results.

I've also tried using the APIs Explorer (Prediction, v1.5) and updating ~300 times that way. There's still no difference in my results. Those updates were also "successful".

200 OK
{
"kind": "prediction#training",
"id": "mymodel",
"selfLink": "https://www.googleapis.com/prediction/v1.5/trainedmodels/mymodel"
}

I am quite sure that the model is receiving these updates. get and analyze both show that the model has numberInstances": "2024". Oddly, though, list shows that the model has "numberInstances": "406".

At this point, I don't know what could be causing this issue.

Welcome to Stack Overflow. It's not necessary, or even standard, to put headers like "**Background**", "**Problem**", etc., in your question. Instead, write it clearly and they won't be needed. In general, format your question like the majority of others and you'll do well. — the Tin Man, Jan 05 '15 at 18:36
Mind answering your own question with the solution? http://stackoverflow.com/help/self-answer — Joshua Grosso Reinstate CMs, Jan 25 '16 at 01:44
FYI, since April 2018 the Cloud Prediction API was shut down. That serve has been replaced by Cloud Machine Learning Engine, https://cloud.google.com/ml-engine/. — Jochem Schulenklopper, Sep 17 '18 at 07:25
Perhaps asking for the obvious case, but what is the value of `MODEL_ID` in your first code snippet? Are you sure that your retrieving the (updated) predictions from the same model you're updating? — Jochem Schulenklopper, Sep 17 '18 at 07:34

score 1 · Answer 1 · answered May 01 '19 at 21:58

2019 Update

Based on the comment from Jochem Schulenklopper that the API was shut down in April 2018.

Developers who choose to move to the Google Cloud Machine Learning Engine will have to recreate their existing Prediction API models.

Machine Learning API examples: https://github.com/GoogleCloudPlatform/cloudml-samples

Update has no effect on scores (Prediction API)

1 Answers1