2

According to the H2O documentation, the threshold used at prediction is the max F1 threshold from train. The performance function, 

h2o.performance(model, newdata = test)

actually run the prediction on the test set in order to compute the confusion matrix.

Strangely I am getting different confusion matrix while predicting the same test set using :

h2o.predict(object, newdata=test).

It means that h2o.performance() is using a different threshold from h2o.predict(). I am wondering how can i dictate the threshold upon prediction.

Nakx
  • 1,460
  • 1
  • 23
  • 32
ronencozen
  • 1,991
  • 1
  • 15
  • 26

1 Answers1

2

H2O is using max F1 threshold for both h2o.performance() and h2o.predict(). The difference is what dataset it will use to estimate the max F1 threshold.

h2o.predict() will use the threshold it selected during training. It uses different max F1 thresholds depending on how the model was trained. Basically:

  • If you only have training data - the max F1 threshold comes from the train data model.
  • If there is validation data during training - the max F1 threshold comes from the validation data model.

This is explained in the documentation and also on stackoverflow. Depending on if you had validation data during training, you will see the max F1 threshold to be determined by your training or validation dataset.

h2o.performance() will take the model and newdata and calculate what threshold will give the highest F1 for the new data. In your case, test is being used to calculate max F1 threshold.

Neema Mashayekhi
  • 902
  • 4
  • 5