1

I am looking at these two questions and documentation:

Whats the output for Keras categorical_accuracy metrics?

Categorical crossentropy need to use categorical_accuracy or accuracy as the metrics in keras?

https://keras.io/api/metrics/probabilistic_metrics/#categoricalcrossentropy-class

For classification of X-Rays images I (15 classes) I do:

# Compile a model
model1.compile(optimizer = 'adam', loss = 'categorical_crossentropy', 
metrics = ['accuracy']) 

# Fit the model
history1 = model1.fit_generator(train_generator, epochs = 10, 
steps_per_epoch = 10, verbose = 1, validation_data = valid_generator)

My model works and I have an output:

But I am not sure how to add validation accuracy here to compare results and avoid over/underfitting.

double-beep
  • 5,031
  • 17
  • 33
  • 41
Anakin Skywalker
  • 2,400
  • 5
  • 35
  • 63

1 Answers1

2

I hope the following can help you:

The use of "categorical_crossentropy" tells me that your labels are a one hot encoding over different classes.

Let's say you have 15 classes, the correct prediction would be a vector with 14 zeros, and a one at the corresponding index. In this context "accuracy" will be very high as your model will be correctly predicting mostly zero everywhere, so the accuracy should easily be at least 13/15 = 0.86.

A more suitable metric would be "categorical_accuracy" which will give you 1 if the model predicts the correct index, and else 0.

If you have a validation "categorical_accuracy" better than 1/15 = 0.067 (assuming your class are correctly balanced), your model is better than random.

You can find a list of metrics at keras metrics.

Leo
  • 91
  • 3
  • Thanks! What would you recommend if I have 60000 data points with 14 classes of medical findings (some problems) and 1 class as No finding. Data is balanced 50%-50% approximately in this competition https://www.kaggle.com/c/vinbigdata-chest-xray-abnormalities-detection – Anakin Skywalker Feb 15 '21 at 22:27
  • 1
    In this context it would be interesting to separate these two cases. If you have enough time, perhaps you could try to create a custom accuracy metric for each of these two cases [see here](https://keras.io/api/metrics/#creating-custom-metrics). If not, your model will be better than random when `categorical_accuracy > 0.5`. – Leo Feb 15 '21 at 22:37