Stack Overflow
Stack Overflow

Questions tagged [regularized]

Regularization involves introducing additional information in order to solve an ill-posed problem or to prevent over-fitting by shrinking the parameters of the model stay close to zero

In mathematics and statistics, particularly in the fields of machine learning and inverse problems, regularization involves introducing additional information in order to solve an ill-posed problem or to prevent overfitting. This information is usually of the form of a penalty for complexity, such as restrictions for smoothness or bounds on the vector space norm.

From http://en.wikipedia.org/wiki/Regularization_%28mathematics%29

<

195 questions
109
votes
8 answers

L1/L2 regularization in PyTorch

How do I add L1/L2 regularization in PyTorch without manually computing it?
Wasi Ahmad
  • 35,739
  • 32
  • 114
  • 161
67
votes
3 answers

TensorFlow - regularization with L2 loss, how to apply to all weights, not just last one?

I am playing with a ANN which is part of Udacity DeepLearning course. I have an assignment which involves introducing generalization to the network with one hidden ReLU layer using L2 loss. I wonder how to properly introduce it so that ALL weights…
Maksim Khaitovich
  • 4,742
  • 7
  • 39
  • 70
31
votes
1 answer

RNN Regularization: Which Component to Regularize?

I am building an RNN for classification (there is a softmax layer after the RNN). There are so many options for what to regularize and I am not sure if to just try all of them, would the effect be the same? which components do I regularize for what…
user1581390
  • 1,900
  • 5
  • 25
  • 38
26
votes
1 answer

What is regularization loss in tensorflow?

When training an Object Detection DNN with Tensorflows Object Detection API it's Visualization Plattform Tensorboard plots a scalar named regularization_loss_1 What is this? I know what regularization is (to make the Network good at generalizing…
gustavz
  • 2,964
  • 3
  • 25
  • 47
25
votes
3 answers

TensorFlow - introducing both L2 regularization and dropout into the network. Does it makes any sense?

I am currently playing with ANN which is part of Udactity DeepLearning course. I successful built and train network and introduced the L2 regularization on all weights and biases. Right now I am trying out the dropout for hidden layer in order to…
Maksim Khaitovich
  • 4,742
  • 7
  • 39
  • 70
21
votes
1 answer

Python pandas time series interpolation and regularization

I am using Python Pandas for the first time. I have 5-min lag traffic data in csv format: ... 2015-01-04 08:29:05,271238 2015-01-04 08:34:05,329285 2015-01-04 08:39:05,-1 2015-01-04 08:44:05,260260 2015-01-04 08:49:05,263711 ... There are several…
riccamini
  • 1,161
  • 1
  • 13
  • 29
20
votes
3 answers

Improving a badly conditioned matrix

I have a badly conditioned matrix, whose rcond() is close to zero, and therefore, the inverse of that matrix does not come out to be correct. I have tried using pinv() but that does not solve the problem. This is how I am taking the inverse: X =…
user238469
12
votes
1 answer

Regularization strategy in Keras

I have trying to setup a non-linear regression problem in Keras. Unfortunately, results show that overfitting is occurring. Here is the code, model = Sequential() model.add(Dense(number_of_neurons, input_dim=X_train.shape[1], activation='relu',…
trumee
  • 393
  • 1
  • 4
  • 11
12
votes
4 answers

Regularized logistic regression code in matlab

I'm trying my hand at regularized LR, simple with this formulas in matlab: The cost function: J(theta) = 1/m*sum((-y_i)*log(h(x_i)-(1-y_i)*log(1-h(x_i))))+(lambda/2*m)*sum(theta_j) The gradient: ∂J(theta)/∂theta_0 = [(1/m)*(sum((h(x_i)-y_i)*x_j)]…
Pedro.Alonso
  • 1,007
  • 3
  • 20
  • 41
11
votes
3 answers

Is the Keras implementation of dropout correct?

The Keras implementation of dropout references this paper. The following excerpt is from that paper: The idea is to use a single neural net at test time without dropout. The weights of this network are scaled-down versions of the trained …
user3390629
  • 854
  • 10
  • 17
10
votes
1 answer

Why word2vec doesn't use regularization?

ML models with huge number of parameters will tend to overfit (since they have a large variance). In my opinion, word2vec is one such models. One of the ways to reduce the model variance is to apply a regularization technique, which is very common…
Tural Gurbanov
  • 742
  • 2
  • 7
  • 27
9
votes
1 answer

Too strong regularization for an autoencoder (Keras autoencoder tutorial code)

I'm using this tutorial about autoencoders: https://blog.keras.io/building-autoencoders-in-keras.html All the code is working, however the performance is very bad (the results are blurred) when I set 10e-5 for the regularization parameter, which is…
ahstat
  • 539
  • 1
  • 7
  • 16
9
votes
3 answers

Why there is the need of using regularization in machine learning problems?

This might seems a stupid question, but I just can't come up with a reasonable answer. It is said that regularization can help us obtain simple models over complex ones to avoid over-fitting. But for a linear classification problem: f(x) = Wx The…
Demonedge
  • 1,363
  • 4
  • 18
  • 33
7
votes
2 answers

How inverting the dropout compensates the effect of dropout and keeps expected values unchanged?

I'm learning regularization in Neural networks from deeplearning.ai course. Here in dropout regularization, the professor says that if dropout is applied, the calculated activation values will be smaller then when the dropout is not applied (while…
Kaushal28
  • 5,377
  • 5
  • 41
  • 72
6
votes
1 answer

How to apply kernel regularization in a custom layer in Keras/TensorFlow?

Consider the following custom layer code from a TensorFlow tutorial: class MyDenseLayer(tf.keras.layers.Layer): def __init__(self, num_outputs): super(MyDenseLayer, self).__init__() self.num_outputs = num_outputs def build(self,…
user79983
  • 75
  • 1
  • 7
1
2 3
…
12 13