Stack Overflow
Stack Overflow

Questions tagged [dropout]

Dropout is a technique to reduce overfitting during the training phase of a neural network.

Dropout is a regularization technique for reducing overfitting in neural networks by preventing complex co-adaptations on training data. The term "dropout" refers to dropping out units (both hidden and visible) in a neural network.

215 questions
99
votes
2 answers

Keras: the difference between LSTM dropout and LSTM recurrent dropout

From the Keras documentation: dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the inputs. recurrent_dropout: Float between 0 and 1. Fraction of the units to drop for the linear transformation of the…
Alonzorz
  • 2,113
  • 4
  • 18
  • 21
66
votes
3 answers

Pytorch: nn.Dropout vs. F.dropout

There are two ways to perform dropout: torch.nn.Dropout torch.nn.functional.Dropout I ask: Is there a difference between them? When should I use one over the other? I don't see any performance difference when I switched them around.
CutePoison
  • 4,679
  • 5
  • 28
  • 63
46
votes
2 answers

How to understand SpatialDropout1D and when to use it?

Occasionally I see some models are using SpatialDropout1D instead of Dropout. For example, in the Part of speech tagging neural network, they use: model = Sequential() model.add(Embedding(s_vocabsize, EMBED_SIZE, …
Raven Cheuk
  • 2,903
  • 4
  • 27
  • 54
29
votes
3 answers

PyTorch - How to deactivate dropout in evaluation mode

This is the model I defined it is a simple lstm with 2 fully connect layers. import copy import torch import torch.nn as nn import torch.nn.functional as F import torch.optim as optim class mylstm(nn.Module): def __init__(self,input_dim,…
Tommy Yu
  • 1,080
  • 3
  • 11
  • 30
22
votes
1 answer

Using Dropout with Keras and LSTM/GRU cell

In Keras you can specify a dropout layer like this: model.add(Dropout(0.5)) But with a GRU cell you can specify the dropout as a parameter in the constructor: model.add(GRU(units=512, return_sequences=True, dropout=0.5, …
BigBadMe
  • 1,754
  • 1
  • 19
  • 27
20
votes
2 answers

Implementing dropout from scratch

This code attempts to utilize a custom implementation of dropout : %reset -f import torch import torch.nn as nn # import torchvision # import torchvision.transforms as transforms import torch import torch.nn as nn import torch.utils.data as…
blue-sky
  • 51,962
  • 152
  • 427
  • 752
18
votes
1 answer

ReLu and Dropout in CNN

I am studying Convolutional Neural Networks. I am confused about some layers in CNN. Regarding ReLu... I just know that it is the sum of an infinite logistic function, but ReLu doesn't connect to any upper layers. Why do we need ReLu, and how does…
user3783676
  • 429
  • 2
  • 6
  • 16
14
votes
2 answers

Dropout layer before or after LSTM. What is the difference?

Suppose that we have an LSTM model for time series forecasting. Also, this is a multivariate case, so we're using more than one feature for training the model. ipt = Input(shape = (shape[0], shape[1]) x = Dropout(0.3)(ipt) ## Dropout before…
Eghbal
  • 3,892
  • 13
  • 51
  • 112
9
votes
1 answer

Tensorflow LSTM Dropout Implementation

How specifically does tensorflow apply dropout when calling tf.nn.rnn_cell.DropoutWrapper() ? Everything I read about applying dropout to rnn's references this paper by Zaremba et. al which says don't apply dropout between recurrent connections.…
beeCwright
  • 370
  • 1
  • 3
  • 12
8
votes
2 answers

Which PyTorch modules are affected by model.eval() and model.train()?

The model.eval() method modifies certain modules (layers) which are required to behave differently during training and inference. Some examples are listed in the docs: This has [an] effect only on certain modules. See documentations of particular…
iacob
  • 20,084
  • 6
  • 92
  • 119
8
votes
1 answer

correct order for SpatialDropout2D, BatchNormalization and activation function?

For a CNN architecture I want to use SpatialDropout2D layer instead of Dropout layer. Additionaly I want to use BatchNormalization. So far I had always set the BatchNormalization directly after a Convolutional layer but before the activation…
Code Now
  • 711
  • 2
  • 9
  • 20
8
votes
1 answer

Keras LSTM: dropout vs recurrent_dropout

I realize this post is asking a similar question to this. But I just wanted some clarification, preferably a link to some kind of Keras documentation that says the difference. In my mind, dropout works between neurons. And recurrent_dropout works…
Oliver Crow
  • 334
  • 6
  • 14
8
votes
2 answers

Keras Dropout with noise_shape

I have a question about Keras function Dropout with the argument of noise_shape. Question 1: What's the meaning of if your inputs have shape (batch_size, timesteps, features) and you want the dropout mask to be the same for all timesteps, you can…
yihao.fu
  • 143
  • 2
  • 5
8
votes
1 answer

Where to add dropout in neural network?

I have seen description about the dropout in different parts of the neural network: dropout in the weight matrix, dropout in the hidden layer after the matrix multiplication and before relu, dropout in the hidden layer after the relu, and…
DiveIntoML
  • 2,347
  • 2
  • 20
  • 36
7
votes
2 answers

How inverting the dropout compensates the effect of dropout and keeps expected values unchanged?

I'm learning regularization in Neural networks from deeplearning.ai course. Here in dropout regularization, the professor says that if dropout is applied, the calculated activation values will be smaller then when the dropout is not applied (while…
Kaushal28
  • 5,377
  • 5
  • 41
  • 72
1
2 3
…
14 15