What should I do to get low average loss?

Question

I'm an student in hydraulic engineering, working on a neural network in my internship so it's something new for me. I created my neural network but it gives me a high loss and I don't know what is the problem ... you can see the code :

def create_model():
    model = Sequential()
    # Adding the input layer
    model.add(Dense(26,activation='relu',input_shape=(n_cols,)))
    # Adding the hidden layer
    model.add(Dense(60,activation='relu'))
    model.add(Dense(60,activation='relu'))
    model.add(Dense(60,activation='relu'))
    # Adding the output layer
    model.add(Dense(2))
    # Compiling the RNN
    model.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])
    return model

kf = KFold(n_splits = 5, shuffle = True)
model = create_model()
scores = []
for i in range(5):
    result = next(kf.split(data_input), None)
    input_train = data_input[result[0]]
    input_test = data_input[result[1]]
    output_train = data_output[result[0]]
    output_test = data_output[result[1]]
    # Fitting the RNN to the Training set
    model.fit(input_train, output_train, epochs=5000, batch_size=200 ,verbose=2)
    predictions = model.predict(input_test) 
    scores.append(model.evaluate(input_test, output_test))

print('Scores from each Iteration: ', scores)
print('Average K-Fold Score :' , np.mean(scores))

And whene I execute my code, the result is like :

Scores from each Iteration:  [[93.90406122928908, 0.8907562990148529], [89.5892979597845, 0.8907563030218878], [81.26530176050522, 0.9327731132507324], [56.46526102659081, 0.9495798339362905], [54.314151876112994, 0.9579831877676379]]
Average K-Fold Score : 38.0159922589274

Can anyone help me please ? how could I do to make the loss low ?

Answer 1

There are several issues, both with your questions and with your code...

To start with, in general we cannot say that an MSE loss of X value is low or high. Unlike the accuracy in classification problems which is by definition in [0, 1], the loss is not similarly bounded, so there is no general way of saying that a particular value is low or high, as you imply here (it always depends on the specific problem).

Having clarified this, let's go to your code.

First, judging from your loss='mean_squared_error', it would seem that you are in a regression setting, in which accuracy is meaningless; see What function defines accuracy in Keras when the loss is mean squared error (MSE)?. You have not shared what exact problem you are trying to solve here, but if it is indeed a regression one (i.e. prediction of some numeric value), you should get rid of metrics=['accuracy'] in your model compilation, and possibly change your last layer to a single unit, i.e. model.add(Dense(1)).

Second, as your code currently is, you don't actually fit independent models from scratch in each of your CV folds (which is the very essence of CV); in Keras, model.fit works cumulatively, i.e. it does not "reset" the model each time it is called, but it continues fitting from the previous call. That's exactly why if you see your scores, it is evident that the model is significantly better in the later folds (which already gives a hint for improving: add more epochs). To fit independent models as you should do for a proper CV, you should move create_model() inside the for loop.

Third, your usage of np.mean() here is again meaningless, as you average both the loss and the accuracy (i.e. apples with oranges) together; the fact that from 5 values of loss between 54 and 94 you end up with an "average" of 38 should have already alerted you that you are attempting something wrong. Truth is, if you dismiss the accuracy metric, as argued above, you would not have this problem here.

All in all, here is how it seems that your code should be in principle (but again, I have not the slightest idea of the exact problem you are trying to solve, so some details might be different):

def create_model():
    model = Sequential()
    # Adding the input layer
    model.add(Dense(26,activation='relu',input_shape=(n_cols,)))
    # Adding the hidden layer
    model.add(Dense(60,activation='relu'))
    model.add(Dense(60,activation='relu'))
    model.add(Dense(60,activation='relu'))
    # Adding the output layer
    model.add(Dense(1))  # change to 1 unit
    # Compiling the RNN
    model.compile(optimizer='adam', loss='mean_squared_error') # dismiss accuracy
    return model

kf = KFold(n_splits = 5, shuffle = True)
scores = []
for i in range(5):
    result = next(kf.split(data_input), None)
    input_train = data_input[result[0]]
    input_test = data_input[result[1]]
    output_train = data_output[result[0]]
    output_test = data_output[result[1]]
    # Fitting the RNN to the Training set
    model = create_model()  # move create_model here
    model.fit(input_train, output_train, epochs=10000, batch_size=200 ,verbose=2)  # increase the epochs
    predictions = model.predict(input_test) 
    scores.append(model.evaluate(input_test, output_test))

print('Loss from each Iteration: ', scores)
print('Average K-Fold Loss :' , np.mean(scores))