-2

I am training LSTM model with some financial data. Can not disclose details of data as it is real trade data. The issues which I am facing is that while training Keras prints out the logs with info related to train and testing loss, accuracy. In these logs, my testing accuracy is 56% and sometimes around it. But for evaluations, I have created a simple function which makes a prediction on the test data and then using sklearn accuracy score prints out the accuracy of the predictions. Now, this accuracy is 24% and sometimes worse than this. What could be the reason for it? I am 100% sure that test data is the same and there is no bug in my code. What could I do to get good results? I already have tried the tunning learning rate, model architecture, layers, optimizers, gradient clipping etc but still, I am getting the same behavior.

I am adding images of the logs and the confusion matrix. enter image description here enter image description here

Urvish
  • 643
  • 3
  • 10
  • 19
  • where did the confusion matrix comes from? sklearn? how about the same result of keras? – skywalkerytx Mar 12 '19 at 11:05
  • yes, the confusion matrix is from sklearn. – Urvish Mar 12 '19 at 11:13
  • kinda wanna see the confusion matrix of Keras – skywalkerytx Mar 12 '19 at 13:33
  • Accuracy (and consequently confusion matrix) is **not defined** for regression problems, such as yours; see answer in [What function defines accuracy in Keras when the loss is mean squared error (MSE)?](https://stackoverflow.com/questions/48775305/what-function-defines-accuracy-in-keras-when-the-loss-is-mean-squared-error-mse/48788577#48788577) – desertnaut Mar 12 '19 at 13:42
  • Accuracy ad confusion matrix is not for regression problem in my code either. What I am trying to do it predict the volume and next action of a trader as buying or selling. The problem is mainly with the action output accuracy in the logs and in confusion matrix. – Urvish Mar 13 '19 at 04:27

1 Answers1

0

I am 100% sure that test data is the same and there is no bug in my code.

Correct me if I'm wrong: If I understand correctly, you have split your data into 2 parts: train and test. You are performing validation on your test data using Keras metrics and then also validating with sklearn which returns different results. You seem to have a regression problem with a custom loss/ metric function.

Have you considered that you might have different accuracy calculation formulas at hand? For example, one formula could be min(prediction, target)/max(prediction, target) while another is 1-(abs(prediction - target)/target)

tYAt1YBWFY
  • 304
  • 2
  • 8
  • No, I did not think that way. But, even if the accuracy formulas are different should not they return the same number or at least something close to one another? also previously I was doing the same for some other dataset and in that data, I was getting the same accuracy in both the logs and function using sklearn. – Urvish Mar 12 '19 at 08:00
  • Try scaling your output linearly by a factor of 10 and see how that affects your metrics. For regression problems, it's not so easy to calculate accuracy in percent and metrics often represent a different value. – tYAt1YBWFY Mar 12 '19 at 08:14
  • 1
    The above accuracy is for classification problem. It is not for the regression problem. – Urvish Mar 12 '19 at 08:32