Keras validation accuracy won't go over 75% and prediction performs bad

Question

I'm working on this dataset to make pollution prediction (NO2) using keras library. I made interpolation on missing data, one hot encoding on wind direction. Divided records on train/validation and test set an gave them to keras through a batch generator after doing Normalization through MinMaxScaler

This are some model I used but none of them seems to go over 75% of validation accuracy and prediction made are really bad:

#### MODEL TYPE DECLARATION AND CONFIGURATION 
######################################

code_name = {-1: "linear", 0: "ann_base_single", 1 : "ann_base_multi", 
             2 : "gru_single_layer", 3 : "gru_single_layer_w_dropout", 4 : "gru_multi_layer", 5 : "gru_multi_layer_2",
             6 : "lstm_single_layer", 7 : "lstm_single_layer_w_dropout", 8: "lstm_multi_layer", 9 : "lstm_multi_layer_2"}
model_type = 20

model = Sequential()
if model_type == -1:
    model.add(layers.Flatten(input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.Dense(num_y_signals, activation='linear'))
elif model_type == 0:
    model.add(layers.Flatten(input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.Dense(hidden_layer_size, activation='relu'))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 1:
    model.add(layers.Flatten(input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.Dense(hidden_layer_size, activation='relu'))
    model.add(layers.Dense(hidden_layer_size, activation='relu'))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 2:
    model.add(layers.GRU(hidden_layer_size, activation='relu', input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 3:
    model.add(layers.GRU(hidden_layer_size, activation='relu', input_shape=(sequence_length, x_data.shape[1])))
    model.add(Dropout(0.2))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 4:
    model.add(layers.GRU(64, activation='relu', return_sequences=True, input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.GRU(32, activation='relu'))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 5:
    model.add(layers.GRU(64, activation='relu', return_sequences=True, input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.GRU(32, activation='relu', return_sequences=True))
    model.add(layers.GRU(16, activation='relu'))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 6:
    model.add(layers.LSTM(hidden_layer_size, activation='relu', input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 7:
    model.add(layers.LSTM(hidden_layer_size, activation='relu', input_shape=(sequence_length, x_data.shape[1])))
    model.add(Dropout(0.2))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 8:
    model.add(layers.LSTM(64, activation='relu', return_sequences=True, input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.LSTM(32, activation='relu'))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))
elif model_type == 9:
    model.add(layers.LSTM(64, activation='relu', return_sequences=True, input_shape=(sequence_length, x_data.shape[1])))
    model.add(layers.LSTM(32, activation='relu', return_sequences=True))
    model.add(layers.LSTM(16, activation='relu'))
    model.add(layers.Dense(num_y_signals, activation='sigmoid'))

model.compile(optimizer=Adam(), loss='mae', metrics=[metrics.mae, 'accuracy'])
model.summary()

timenow = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S_")
run_name = '{7}_Nodes_{0}-Units_{1}-shift_steps_{2}-aq_parameter_{3}-train_split_{4}-batch_size_{5}-sequence_length_{6}'.format(code_name[model_type], hidden_layer_size, shift_steps, aq_parameter, train_split, batch_size, sequence_length, timenow)
path_checkpoint = 'checkpoints\\{0}.keras'.format(run_name)

#### CALLBACK DECLARATION #######################################

callback_checkpoint = ModelCheckpoint(filepath=path_checkpoint,
                                      monitor='val_loss',
                                      verbose=1,
                                      save_weights_only=True,
                                      save_best_only=True)

callback_early_stopping = EarlyStopping(monitor='val_loss',
                                        patience=4, verbose=1)

callback_reduce_lr = ReduceLROnPlateau(monitor='val_loss',
                                       factor=0.1,
                                       min_lr=1e-5,
                                       patience=3,
                                       verbose=1)

callbacks = [callback_checkpoint,
             callback_early_stopping,
             callback_reduce_lr,
             callback_reset_states]


#### FITTING DATA INTO MODEL

    history = model.fit_generator(generator=train_generator,
                              steps_per_epoch=int(train_data_size / batch_size),
                              epochs=epochs,
                              validation_data=validation_data,
                              callbacks = callbacks,
                              shuffle= False)

Followed this tutorial.

enter enter enter

I would like to know, how to improve this model so it can be over 75% val_acc and made accurate prevision... I'm new to machine learning and maybe I made some mistakes on data or something else — Giacomo Tontini, Aug 09 '18 at 08:13
@desertnaut how are you supposed to understand a neural network problem without having a full panorama of it? People usually downvote due to poor info, not the opposite — Giacomo Tontini, Aug 09 '18 at 08:56
Please *do read the link* (especially *Why this is a problem* & *Why this is worth a downvote*), as well as [How to create a **Minimal**, Complete, and Verifiable example](https://stackoverflow.com/help/mcve); some hints: 1) which one of your 22 (!) models produces these graphs? 2) is it `stateful` or not? 3) are the Tensorboard callbacks really necessary for understanding your *problem*? SO does not work by just throwing all your code into the question, and it requires some additional steps from your side, so as to help us in order to help *you*... — desertnaut, Aug 09 '18 at 09:09
And how can you combine MAE (a regression metric) with accuracy (a classification one)? Is your problem a regression or a classification one? It seems to be the former, in which case **accuracy is meaningless**... — desertnaut, Aug 09 '18 at 09:16
1) I wrote so many model because all of them (more or less) produces the problem. I don't know how to determine which one is better 2) Stateful need to be set true for the model with stateful model enabled and to switch between fit_generator calls (with generator even for validating or not) 3) tensorboard callback could be removed from the code section (removed) 4) it's a regression problem since I need to predict a NO2 value (numerical and not categorical) could you explain why these two metrics could not be used together? People on discussion uses both — Giacomo Tontini, Aug 09 '18 at 10:29
Which people on which discussion? Before proceeding to rather complicated ML attempts, like the one you present here, pls *do* take some time to **familiarize yourself with the fundamental concepts**; see my answer at [What function defines accuracy in Keras when the loss is mean squared error (MSE)?](https://stackoverflow.com/questions/48775305/what-function-defines-accuracy-in-keras-when-the-loss-is-mean-squared-error-mse/48788577#48788577) (applicable for MAE, too), as well as the comments in the OP there... — desertnaut, Aug 09 '18 at 10:56

score 0 · Answer 1 · answered Aug 09 '18 at 08:37

0

Your code is quite comprehensive , but isn't analytical enough , for you to pinpoint what is exactly is causing your prediction to stay at the range of 75%.

Try to implement some callback functions of keras , so that you can store the model at different stages , and get to know how the performance changes through each layer.

Keras functionality of callback

Model Checkpoint Keras

You will have to extend your code in the Callack Declaration section, taking into account 22 types of models being compared, and check how each model pans out.

answered Aug 09 '18 at 08:37

abunickabhi

558
2
9
31

Since I'm already using this callback from the beginnin I can't face a way to compare all the models at the same time... Are you suggesting to train on a while{} all the model and then compare checkpoints? – Giacomo Tontini Aug 09 '18 at 08:53
Yes trying to compare the checkpoints ,by inserting the models inside a while{} , might be the solution. – abunickabhi Aug 09 '18 at 09:18

Keras validation accuracy won't go over 75% and prediction performs bad

1 Answers1