0

so i am fairly new at this. working on a model to predict a time series problem, and it overfits quickly after around epoch 25, i did some dropout and reduces the network complexity, but still the model can't resist overfitting after a while. also the acc and val_acc are frozen from the beginning to the end. here is the network architecture:

regressor = Sequential()

regressor.add(CuDNNGRU(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))
regressor.add(Dropout(0.3))

regressor.add(CuDNNGRU(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))

regressor.add(CuDNNGRU(units = 50, return_sequences =False))
regressor.add(Dropout(0.3))

# i've intentionally commented out this layer to reduce complexity
# regressor.add(CuDNNGRU(units = 50))
# regressor.add(Dropout(0.2))

regressor.add(Dense(units = 1))

regressor.compile(optimizer = 'adam', loss = 'mean_squared_error', metrics=['accuracy'])

history = regressor.fit(X_train, y_train, epochs = 60, batch_size = 64, validation_split=0.2)

and this is basically how the epochs unravel:

Train on 127936 samples, validate on 31984 samples
Epoch 1/60
127936/127936 [==============================] - 99s 772us/step - loss: 0.0015 - acc: 7.8164e-06 - val_loss: 0.0012 - val_acc: 3.1266e-05
Epoch 2/60
127936/127936 [==============================] - 98s 763us/step - loss: 3.1164e-04 - acc: 7.8164e-06 - val_loss: 8.9437e-04 - val_acc: 3.1266e-05
Epoch 3/60
127936/127936 [==============================] - 97s 762us/step - loss: 2.3211e-04 - acc: 7.8164e-06 - val_loss: 0.0010 - val_acc: 3.1266e-05
Epoch 4/60
127936/127936 [==============================] - 98s 765us/step - loss: 2.2029e-04 - acc: 7.8164e-06 - val_loss: 0.0016 - val_acc: 3.1266e-05
Epoch 5/60
127936/127936 [==============================] - 97s 760us/step - loss: 2.1758e-04 - acc: 7.8164e-06 - val_loss: 0.0028 - val_acc: 3.1266e-05
Epoch 6/60
127936/127936 [==============================] - 98s 765us/step - loss: 2.1232e-04 - acc: 7.8164e-06 - val_loss: 0.0016 - val_acc: 3.1266e-05
Epoch 7/60
127936/127936 [==============================] - 97s 761us/step - loss: 2.1088e-04 - acc: 7.8164e-06 - val_loss: 0.0012 - val_acc: 3.1266e-05
Epoch 8/60
127936/127936 [==============================] - 97s 760us/step - loss: 2.0391e-04 - acc: 7.8164e-06 - val_loss: 0.0012 - val_acc: 3.1266e-05
Epoch 9/60
127936/127936 [==============================] - 97s 761us/step - loss: 2.0780e-04 - acc: 7.8164e-06 - val_loss: 6.3458e-04 - val_acc: 3.1266e-05
Epoch 10/60
127936/127936 [==============================] - 97s 762us/step - loss: 2.0697e-04 - acc: 7.8164e-06 - val_loss: 5.3828e-04 - val_acc: 3.1266e-05
Epoch 11/60
127936/127936 [==============================] - 98s 763us/step - loss: 2.0546e-04 - acc: 7.8164e-06 - val_loss: 4.7978e-04 - val_acc: 3.1266e-05
Epoch 12/60
127936/127936 [==============================] - 98s 763us/step - loss: 2.0410e-04 - acc: 7.8164e-06 - val_loss: 3.6436e-04 - val_acc: 3.1266e-05
Epoch 13/60
127936/127936 [==============================] - 98s 763us/step - loss: 2.0250e-04 - acc: 7.8164e-06 - val_loss: 3.5326e-04 - val_acc: 3.1266e-05
Epoch 14/60
127936/127936 [==============================] - 97s 758us/step - loss: 2.0288e-04 - acc: 7.8164e-06 - val_loss: 4.8453e-04 - val_acc: 3.1266e-05
Epoch 15/60
127936/127936 [==============================] - 98s 763us/step - loss: 2.0580e-04 - acc: 7.8164e-06 - val_loss: 0.0014 - val_acc: 3.1266e-05
Epoch 16/60
127936/127936 [==============================] - 97s 760us/step - loss: 2.0156e-04 - acc: 7.8164e-06 - val_loss: 0.0011 - val_acc: 3.1266e-05
Epoch 17/60
127936/127936 [==============================] - 97s 762us/step - loss: 2.0055e-04 - acc: 7.8164e-06 - val_loss: 0.0012 - val_acc: 3.1266e-05
Epoch 18/60
127936/127936 [==============================] - 97s 759us/step - loss: 2.0162e-04 - acc: 7.8164e-06 - val_loss: 0.0013 - val_acc: 3.1266e-05
Epoch 19/60
127936/127936 [==============================] - 97s 761us/step - loss: 1.9856e-04 - acc: 7.8164e-06 - val_loss: 7.1617e-04 - val_acc: 3.1266e-05
Epoch 20/60
127936/127936 [==============================] - 97s 758us/step - loss: 2.0146e-04 - acc: 7.8164e-06 - val_loss: 8.7160e-04 - val_acc: 3.1266e-05
Epoch 21/60
127936/127936 [==============================] - 97s 761us/step - loss: 2.0139e-04 - acc: 7.8164e-06 - val_loss: 0.0017 - val_acc: 3.1266e-05
Epoch 22/60
127936/127936 [==============================] - 97s 760us/step - loss: 2.0001e-04 - acc: 7.8164e-06 - val_loss: 0.0013 - val_acc: 3.1266e-05
Epoch 23/60
127936/127936 [==============================] - 97s 761us/step - loss: 2.0003e-04 - acc: 7.8164e-06 - val_loss: 7.9431e-04 - val_acc: 3.1266e-05
Epoch 24/60
127936/127936 [==============================] - 98s 763us/step - loss: 1.9823e-04 - acc: 7.8164e-06 - val_loss: 0.0011 - val_acc: 3.1266e-05
Epoch 25/60
127936/127936 [==============================] - 98s 762us/step - loss: 1.9902e-04 - acc: 7.8164e-06 - val_loss: 0.0012 - val_acc: 3.1266e-05
Epoch 26/60
127936/127936 [==============================] - 97s 762us/step - loss: 1.9857e-04 - acc: 7.8164e-06 - val_loss: 0.0012 - val_acc: 3.1266e-05
Epoch 27/60
127936/127936 [==============================] - 97s 761us/step - loss: 1.9652e-04 - acc: 7.8164e-06 - val_loss: 0.0020 - val_acc: 3.1266e-05
Epoch 28/60
127936/127936 [==============================] - 97s 760us/step - loss: 1.9940e-04 - acc: 7.8164e-06 - val_loss: 0.0017 - val_acc: 3.1266e-05
Epoch 29/60
127936/127936 [==============================] - 97s 758us/step - loss: 1.9802e-04 - acc: 7.8164e-06 - val_loss: 0.0032 - val_acc: 3.1266e-05
Epoch 30/60
127936/127936 [==============================] - 97s 761us/step - loss: 1.9759e-04 - acc: 7.8164e-06 - val_loss: 0.0018 - val_acc: 3.1266e-05
Epoch 31/60
127936/127936 [==============================] - 97s 757us/step - loss: 1.9716e-04 - acc: 7.8164e-06 - val_loss: 0.0033 - val_acc: 3.1266e-05
Epoch 32/60
127936/127936 [==============================] - 97s 760us/step - loss: 1.9611e-04 - acc: 7.8164e-06 - val_loss: 0.0038 - val_acc: 3.1266e-05
Epoch 33/60
127936/127936 [==============================] - 97s 762us/step - loss: 1.9661e-04 - acc: 7.8164e-06 - val_loss: 0.0066 - val_acc: 3.1266e-05
Epoch 34/60
127936/127936 [==============================] - 98s 762us/step - loss: 1.9186e-04 - acc: 7.8164e-06 - val_loss: 0.0068 - val_acc: 3.1266e-05
Epoch 35/60
127936/127936 [==============================] - 97s 762us/step - loss: 1.9449e-04 - acc: 7.8164e-06 - val_loss: 0.0077 - val_acc: 3.1266e-05

as you can see, after epoch 23 the val_loss started soaring. also why is there ever no chance in acc and val_acc in this model?

Mahdi_J
  • 408
  • 2
  • 15
  • correction of the sentence above: no change* – Mahdi_J Sep 11 '19 at 19:56
  • It looks like your model is pretty much converged after the first epoch, except it hasn’t learned anything. Is your data definitely suitable for this? It’s not weekly lottery numbers or something? But I think I’m interpreting "acc" as I would for classification (where a low accuracy is poor), and really it means your model is very good after one epoch (depending on your data values). Have you seen [this](https://stackoverflow.com/questions/45632549/keras-accuracy-for-my-model-always-0-when-training)? – Pam Sep 12 '19 at 18:05
  • Thank you for the comment @Pam. no its actually the gold/usd data and surprisingly the prediction is not too horrible, i am willing to give on the idea of having val_acc and acc because this isn't exactly a classification issue, but the thing that keeps bothering me most is the fact that no matter how much dropout i add and how much complexity i reduce, it still keeps horribly overfitting after epoch 20. – Mahdi_J Sep 12 '19 at 18:10
  • Hmmm, I’m not overly familiar with regression (more classification), but your set up brought [this](https://www.linkedin.com/pulse/how-use-machine-learning-time-series-forecasting-vegard-flovik-phd/) article to mind. Especially the graphs where he shows that his prediction graph is simply following his actual graph one step behind. I hope you can figure it out. – Pam Sep 12 '19 at 18:30
  • thank you very much @Pam. one more question if you wouldn't mind, if i remove the metrics, all the metrics will disappear except for the loss. then how would i know when my model is starting to overfit?(since i assume one needs the difference between loss and val_loss to find that out). – Mahdi_J Sep 12 '19 at 18:49
  • Don’t remove them, just use one that is more fitting for regression (I.e.MSE? Or another from the post in a previous comment). Over fitting is technically where network parameters adjust too far so that they closely match the training set but don’t generalise to the test or validation set. It could also manifest as increasing validation/test set loss as in your trace. It’s even a good sign that your train/validation sets are disjoint. – Pam Sep 12 '19 at 19:40

0 Answers0