Why my accuracy is always 0.2 in this simple code

Question

I am new in this field and trying to re-run an example LSTM code copied from internet. The accuracy of the LSTM model is always 0.2 but the predicted output is totally correct which means the accuracy should be 1. Could anyone tell me why?

from numpy import array
from keras.models import Sequential, Dense, LSTM

length = 5
seq = array([i/float(length) for i in range(length)])  
print(seq)    
X = seq.reshape(length, 1, 1)    
y = seq.reshape(length, 1)

# define LSTM configuration
n_neurons = length
n_batch = 1000
n_epoch = 1000

# create LSTM
model = Sequential()    
model.add(LSTM(n_neurons, input_shape=(1, 1)))    
model.add(Dense(1))    
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])



# train LSTM
model.fit(X, y, epochs=n_epoch, batch_size=n_batch)#, verbose=2)    
train_loss, train_acc = model.evaluate(X, y)

print('Training set accuracy:', train_acc    
result = model.predict(X, batch_size=n_batch, verbose=0)    
for value in result:
    print('%.1f' % value)

I believe https://stackoverflow.com/a/47522433/5510499 answers your question. — Vadim Shkaberda, Jun 04 '19 at 18:53

score 0 · Answer 1 · edited Jun 04 '19 at 19:21

You are measuring accuracy, but you are training a regressor. This means you are having as output a float number and not a fixed categorical value.

If you change the last print to have 3 decimals of precision (print('%.3f' % value) ) you will see that the predicted values are really close to the ground truth but not exactly the same, therefore the accuracy is low:

For some reason, the accuracy being used (sparse_categorical_accuracy) is considering the 0.0 and 0.039 (or similar) as a hit instead of a miss, so that's why you are getting 20% instead of 0%.

If you change the sequence to not contain zero, you will have 0% accuracy, which is less confusing:

seq = array([i/float(length) for i in range(1, length+1)])

Finally, to correct this, you can use, for example, mae instead of accuracy as the metric, where you will see the error going down:

model.compile(loss='mean_squared_error', optimizer='adam', metrics=['mae'])

Other option would be to switch to a categorical framework (changing your floats to categorical values).

Hope this helps! I will edit the answer if I can dig into why the sparse_categorical_accuracy detects the 0 as a hit and not a miss.

Why my accuracy is always 0.2 in this simple code

1 Answers1