Keras: the difference between LSTM dropout and LSTM recurrent dropout

Question

From the Keras documentation:

dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs.

recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the recurrent state.

Can anyone point to where on the image below each dropout happens?

`for the linear transformation of the inputs` so x_t – Thomas Jungblut Jul 05 '17 at 11:14 — Thomas Jungblut, Jul 05 '17 at 11:14

Michele Tonutti · Accepted Answer · 2019-01-13T11:06:06.290

99

I suggest taking a look at (the first part of) this paper. Regular dropout is applied on the inputs and/or the outputs, meaning the vertical arrows from x_t and to h_t. In your case, if you add it as an argument to your layer, it will mask the inputs; you can add a Dropout layer after your recurrent layer to mask the outputs as well. Recurrent dropout masks (or "drops") the connections between the recurrent units; that would be the horizontal arrows in your picture.

This picture is taken from the paper above. On the left, regular dropout on inputs and outputs. On the right, regular dropout PLUS recurrent dropout:

(Ignore the colour of the arrows in this case; in the paper they are making a further point of keeping the same dropout masks at each timestep)

edited Jan 13 '19 at 11:06

answered Jul 05 '17 at 14:59

Michele Tonutti

4,298
1
21
22

1

Thanks, @michetonu. The linked paper and your explanation were helpful. Anything you could point to in terms of how to properly use regular dropout with recurrent_dropout in Keras for time series forecasting? Seems some examples out there combine the two kinds of dropout, whiles use just recurrent_dropout. – Kim Miller Jul 30 '18 at 21:14
1

@KimMiller from what I've encountered, there does not seem to be much science behind using one or another, and how much (yet). I tend to optimize both and choose whatever combination works best. – Michele Tonutti Jul 31 '18 at 10:47
14

And there seems to be applying the (regular) dropout= parameter on a layer vs. a dropout layer by itself. What is the effective difference between these methods? – Kim Miller Aug 01 '18 at 17:11

score 17 · Answer 2 · answered Jul 17 '20 at 21:06

Above answer highlights one of the recurrent dropout methods but that one is NOT used by tensorflow and keras. Tensorflow Doc.

Keras/TF refers a recurrent method proposed by Semeniuta et al. Also, check below the image comparing different recurrent dropout methods. The Gal and Ghahramani method which is mentioned in above answer is at second position and Semeniuta method is the right most.

Keras: the difference between LSTM dropout and LSTM recurrent dropout

2 Answers2

Linked