1

I have to build a model in keras. I am really struggling with my actual dataset, hence I am just trying to figure out the basics on a simpler dataset.

model = Sequential([
    Dense(32, input_dim=X_train.shape[1], activation="sigmoid"),
    Dense(2, activation="softmax"),
])
metrics=[ 
    tf.keras.metrics.TruePositives(name="tp"),
    tf.keras.metrics.TrueNegatives(name="tn"),
    tf.keras.metrics.FalseNegatives(name="fn"),
    tf.keras.metrics.FalsePositives(name="fp"),
    tf.keras.metrics.Recall(name="recall"),
    tf.keras.metrics.Precision(name="precision")
]

model.compile(loss="categorical_crossentropy", metrics=metrics, optimizer="sgd")
model.evaluate(X_test, y_test)
evaluation = model.evaluate(X_test, y_test)
for i, m in model.metrics_names:
    print(m, evaluation[i])

This gets printed out:

loss 0.4604386021425058
tp 2965.5
tn 2965.5
fn 531.25
fp 531.25
recall 0.8480753898620605
precision 0.8480753898620605

Something really strange about this results. I believe it is due to using the softmax with two nodes.

y_train looks something like this:

array([[1., 0.],
       [1., 0.],
       [1., 0.],

       [1., 0.]], dtype=float32)

I tried a sigmoid, but then the whole model breaks down, at least here the fitting works.

Is there a way to configure recall and precision so they consider one output notes as Positive?

Borut Flis
  • 15,715
  • 30
  • 92
  • 119
  • There seems to be a bug as you are using `tf.keras.metrics.FalseNegatives(name="fp")` instead of `tf.keras.metrics.FalsePositives(name="fp")` to calculate the false positives. – Flavia Giammarino Aug 08 '21 at 15:06
  • Thanks, I made a typo. But the results remain the same after fixing it. – Borut Flis Aug 08 '21 at 15:17
  • 1
    You can convert your train data into a one-dimensional array. For example [0,1] becomes 1, [1,0] becomes 0. Then you can convert your last dense layer into a Dense(1) and get the desired result. Otherwise, precision&recall is calculated for both class. – Hakan Akgün Aug 08 '21 at 15:46
  • 1
    What do you mean by *whole model breaks down*? Indeed they are like this because of softmax. – Frightera Aug 08 '21 at 15:48
  • @Frighter There is some keras error I don't understand, I will post it later. – Borut Flis Aug 08 '21 at 17:37
  • @HakanAkgün is there any way to split the recall? – Borut Flis Aug 08 '21 at 17:38
  • 1
    You can't use these metrics with 2-way softmax. One way would be the obtaining predictions after training and using sklearn to get recall-precision. – Frightera Aug 08 '21 at 17:48

1 Answers1

1

The only solution in your case is to transform the problem into a one-dimensional one, i.e.

  • Use Dense(1,activation='sigmoid') instead of Dense(2,activation='softmax'); change [0,1] to 0 and [1,0] to 1 as an example.
  • Use binary_crossentropy instead of categorical_crossentropy.

Otherwise, you can implement a special callback to retrieve those metrics (using scikit-learn, like in the example below):

How to get other metrics in Tensorflow 2.0 (not only accuracy)?

Timbus Calin
  • 13,809
  • 5
  • 41
  • 59
  • Thanks. I tried this approach before and something went wrong than. – Borut Flis Aug 09 '21 at 11:59
  • I still don't understand everything though. For example I have 120 examples in the test set. Yet the fp, tp, fn, tn numbers returned are 3064.000000, 33.000000, 551.750000, 68.000000. It looks they are cumulatively adding the numbers??? – Borut Flis Aug 09 '21 at 12:10