why is the accuracy constant but loss does change?

Question

As you can see below i have two functions , get_data() outputs a data frame for the selected asset history and passes it to train_model() every thing works fine but as the model trains the accuracy does not seem to change the loss does go down but the accuracy stays the same after the second epoch ,when training with 1000 epochs the accuracy also does not change

Things i tried changing with this code:

changing unit count for each of the LSTM layers
tried using differnet data frames from different sources ( alpha-vantage )
changing epoch count

unfortunately nothing changed

def train_model( df):

    if not os.path.exists("/py_stuff/"):
        os.makedirs("/py_stuff/")


    checkpoint_filepath ="/py_stuff/check_point"
    weights_checkpoint = "/py_stuff/"


    checkpoint_dir = os.path.dirname(checkpoint_filepath)


    model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(
        filepath=checkpoint_filepath,
        save_weights_only=True,
        monitor='accuracy',
        mode='max',
        save_best_only=True,
        verbose=1)


    dataset_train = df
    training_set = dataset_train.iloc[:, 1:2].values

    sc = MinMaxScaler(feature_range=(0,1))
    training_set_scaled = sc.fit_transform(training_set)

    X_train = []
    y_train = []
    for i in range(100, len(df)):
        X_train.append(training_set_scaled[i-100:i, 0])
        y_train.append(training_set_scaled[i, 0])
    X_train, y_train = np.array(X_train), np.array(y_train)
    X_train = np.reshape(X_train, (X_train.shape[0], X_train.shape[1], 1))


    model = Sequential()
    model.add(LSTM(units = 100, return_sequences = True, input_shape = (X_train.shape[1], 1)))
    model.add(Dropout(0.2))

    model.add(LSTM(units=100 ,  return_sequences=True))
    model.add(Dropout(0.2))
    model.add(LSTM(units=100 , return_sequences=True))
    model.add(Dropout(0.2))
    model.add(LSTM(units=100))
    model.add(Dropout(0.2))
    model.add(Dense(units=1))
    model.compile(optimizer='adam', loss='mean_squared_error' , metrics=['accuracy'])

        ## loading weights 

    try:
        model.load_weights(checkpoint_filepath)
        print ("Weights loaded successfully $$$$$$$ ")
    except:
        print ("No Weights Found !!! ")



    model.fit(X_train,y_train,epochs=50,batch_size=100, callbacks=[model_checkpoint_callback])

    ## saving weights 


    try:
        model.save(checkpoint_filepath)
        model.save_weights(filepath=checkpoint_filepath)

        print ("Saving weights and model done ")

    except OSError as no_model:
        print ("Error saving weights and model !!!!!!!!!!!! ")





def get_data(CHOICE):
        data = yf.download(  # or pdr.get_data_yahoo(...
        # tickers list or string as well
                tickers = CHOICE,

        # use "period" instead of start/end
        # valid periods: 1d,5d,1mo,3mo,6mo,1y,2y,5y,10y,ytd,max
        # (optional, default is '1mo')
                period = "5y",

        # fetch data by interval (including intraday if period < 60 days)
        # valid intervals: 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo
        # (optional, default is '1d')
                interval = "1d",

        # group by ticker (to access via data['SPY'])
        # (optional, default is 'column')
                group_by = 'ticker',

        # adjust all OHLC automatically
        # (optional, default is False)
                auto_adjust = True,

        # download pre/post regular market hours data
        # (optional, default is False)
                prepost = True,

        # use threads for mass downloading? (True/False/Integer)
        # (optional, default is True)
                threads = True,

        # proxy URL scheme use use when downloading?
        # (optional, default is None)
                proxy = None
        )

        dff = pd.DataFrame(data)
        return dff





df = get_data(CHOICE="BTC-USD")



train_model(df)

score 1 · Accepted Answer · answered Jul 19 '20 at 03:31

1

From your loss function, it looks like you have a regression network. Your loss is Mean Squared Error and the metric accuracy does not have any meaning for regression networks. Accuracy metric is only meaningful when used for classification models. So you can remove the metrics=['accuracy'] from your compile code and and use loss value to evaluate your model. So if the loss is decreasing that means your optimizer is successfully training the network.

answered Jul 19 '20 at 03:31

Nima Aghli

484
1
4
10

how to use `model.evaluate()` to show the model accuracy how close does it predict compared to the original value at a specific time? – Sec Team Jul 19 '20 at 07:02
You have to keep a portion of your training data for testing. After you finish training use your test set to evaluate the model. You must compile the model with mean_squared_error or mean_absolute_error before evaluation if you are loading the model from file. Value of mean_squared_error will tell you how the model is doing on testset. For example if you are predicting price of something and use mean_absolute_error and the MAE value is 1 that means the model error on test set is 1$ on average. It is up to you to decide if 1$ error is large or not based on your application. – Nima Aghli Jul 19 '20 at 07:22
consider the two function above in a class , as methods and i save model , df as `self.item = item ` so i can get the data from the object and pass it around the other methods – Sec Team Jul 19 '20 at 14:34
and what do i have to change in`model_checkpoint_callback` because i get this warning `WARNING:tensorflow:Can save best model only with val_loss available, skipping.` – Sec Team Jul 19 '20 at 14:38

score 1 · Answer 2 · answered Jul 19 '20 at 04:27

You are dealing with a regression problem where the accuracy is not defined.

The accuracy is defined as the probability of belonging to a specific class. For example, the probability of the output to be the digit 9. The number of classes is finite (or countable).

In your case, your network outputs a real number. The notion of accuracy does not make any sense in this context.

For example, the probability of your output to be 1.000 for example is 0. Although (and surprisingly!), a probability of zero does not mean that the event will never happen!

Ideally, Keras should return an error saying accuracy not defined.

why is the accuracy constant but loss does change?

2 Answers2