1

In training, Keras categorical_accuracy value is 100%. But that same training data that I've save it's output to a file, shown several (actually quite many) data that classified to wrong class. I've check input file for label, and it's the correct one.

What does categorical_accuracy measure? Is there any better metric to debug a LSTM?

model = Sequential()
model.add(LSTM(64, batch_input_shape=(7763, TimeStep.TIME_STEP + 1, 10), return_sequences=True, activation='relu'))
model.add(LSTM(128, activation='relu', return_sequences=True))
model.add(LSTM(64, activation='relu', return_sequences=True))
model.add(LSTM(32, activation='relu'))
model.add(Dense(3, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=[categorical_accuracy])
history = model.fit(TimeStep.fodder, TimeStep.target, epochs=300, batch_size=7763)





Epoch 400/400

 7763/31052 [======>.......................] - ETA: 1s - loss: 2.7971e-04 - categorical_accuracy: 1.0000
15526/31052 [==============>...............] - ETA: 1s - loss: 3.0596e-04 - categorical_accuracy: 1.0000
23289/31052 [=====================>........] - ETA: 0s - loss: 3.0003e-04 - categorical_accuracy: 1.0000
31052/31052 [==============================] - 2s 78us/step - loss: 2.9869e-04 - categorical_accuracy: 1.0000
Erland Devona
  • 183
  • 3
  • 13
  • Please try to create a [MRE](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) example. We don't have your data, we don't have your code. Here is the [documentation](http://faroit.com/keras-docs/1.2.0/metrics/#categorical_accuracy) about the metric you chose and it should work. – Corentin Limier Jul 11 '19 at 08:43

1 Answers1

0

From the keras github repo we have the categorical_accuracyfunction.

def categorical_accuracy(y_true, y_pred):
    return K.cast(K.equal(K.argmax(y_true, axis=-1),
                          K.argmax(y_pred, axis=-1)),
                  K.floatx())

Here we can see that if the position of the maximum value of y_true is the same as y_pred it returns 1, else 0. A 100% accuracy should indicate that the position of the maximum value of y_true is always the same as y_pred (position here is classes, so always predict the same class).

One possible reason for this could be that you only have one single output (single binary class). hence the position of the maximum value will always be 0 for both y_true and y_pred.

In this is the case use binary_accuracy instead.

KrisR89
  • 1,483
  • 5
  • 11
  • LSTM output have 3 classes – Erland Devona Jul 11 '19 at 08:49
  • your code is not an working example, so cannot be 100% sure. But it seems that the model is predicting correct on the training data. this could be partly because of over-fitting (400 epochs is quite a lot without dropout, and you also have an unusual high number of LSTM layers). Can you split your data into train and test, and add the test data to the validation_data input in `model.fit`. My quess is that the accuracy for this is quite far from 1? – KrisR89 Jul 11 '19 at 09:06
  • yes it's quite far from 1. At first I thought the model do not learn, so I try to tweak number of epoch, layers, and neuron. It does not change much from my original model that's 32, 64 with 100 epoch. – Erland Devona Jul 11 '19 at 09:38
  • Start simple, and go from there. Start with a single LSTM layer, then dropout (eks dropout rate of 0.2), then your output dense layer. Your model.compile is fine. Start with epochs less than 5, then for fine-tuning you can start increasing this. plot the loss and accuracy against the corresponding validation values. And look for [signs](https://keras.rstudio.com/articles/tutorial_overfit_underfit.html) of over-fitting . – KrisR89 Jul 11 '19 at 10:47