5

I have a neural network for regression prediction means that the output is a real value number in range 0 to 1.

I used drop out for all layers and the errors suddenly increased and never converged.

Is drop out usable for regression task? Because if we disregard some nodes, the last layer will have fewer nodes and the predicted value will definitely very different from the actual value. So the back propagated error will be large and the model will be destroyed. Then Why should we use drop out for regression task in neural networks?

Paniz
  • 594
  • 6
  • 19

2 Answers2

1

Because if we disregard some nodes, the last layer will have fewer nodes and the predicted value will definitely very different from the actual value.

You are correct. Hence most frameworks scale up the number of neurons during training (and don't during prediction time). This simple hack is effective and works well for most cases. However, it doesn't work that well for a regression task. It works well where the outputs of activation can be relative to each other (like softmax). In regression the values are absolute and the small differences in "train" and "prediction" setups do cause mild instabilities on occasions.

It is always best to start with a 0 dropout and then increase it slowly to observe what value gives the best result

Allohvk
  • 915
  • 8
  • 14
0

I used drop out for all layers and the errors suddenly increased and never converged.

This also happens when you use too many dropouts, especially in regression tasks. Did you tried reducing dropouts?? Also, dropouts is recommended for those layers which has very high number of trainable parameters. Also consider removing dropouts from last layer and then check once.

Gaurav Sharma
  • 138
  • 2
  • 9