Stack Overflow
Stack Overflow

Questions tagged [learning-rate]

83 questions
25
votes
2 answers

Pytorch Change the learning rate based on number of epochs

When I set the learning rate and find the accuracy cannot increase after training few epochs optimizer = optim.Adam(model.parameters(), lr = 1e-4) n_epochs = 10 for i in range(n_epochs): // some training here If I want to use a step decay:…
Shin Yu Wu
  • 1,129
  • 4
  • 14
  • 23
10
votes
1 answer

PyTorch: Learning rate scheduler

How do I use a learning rate scheduler with the following optimizer? optimizer = torch.optim.Adam(optim_params,betas=(args.momentum, args.beta), weight_decay=args.weight_decay) I have written the following scheduler: scheduler =…
doldnh
  • 312
  • 3
  • 12
8
votes
3 answers

Pytorch schedule learning rate

I am trying to re-implement one paper, which suggests to adjust the learning rate as below: The learning rate is decreased by a factor of the regression value with patience epochs 10 on the change value of 0.0001. Should I use the…
Yan-Jen Huang
  • 305
  • 1
  • 2
  • 8
6
votes
1 answer

Properly set up exponential decay of learning rate in tensorflow

I need to apply an exponential decay of learning rate every 10 epochs. Initial learning rate is 0.000001, and decay factor is 0.95 is this the proper way to set it up? lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay( …
mdnfiras
  • 882
  • 1
  • 15
  • 28
5
votes
1 answer

tf.Keras learning rate schedules—pass to optimizer or callbacks?

I just wanted to set up a learning rate schedule for my first CNN and I found there are various ways of doing so: One can include the schedule in callbacks using tf.keras.callbacks.LearningRateScheduler() One can pass it to an optimizer using…
Manuel Popp
  • 1,003
  • 1
  • 10
  • 33
4
votes
1 answer

Why do we multiply learning rate by gradient accumulation steps in PyTorch?

Loss functions in pytorch use "mean" reduction. So it means that the model gradient will have roughly the same magnitude given any batch size. It makes sense that you want to scale the learning rate up when you increase batch size because your…
off99555
  • 3,797
  • 3
  • 37
  • 49
3
votes
1 answer

How can I set a minimum learning rate in lr_scheduler LambdaLR?

I'm using LambdaLR as a learning rate function: import torch import torch.nn as nn import matplotlib.pyplot as plt model = torch.nn.Linear(2, 1) optimizer = torch.optim.SGD(model.parameters(), lr=0.01) lambda1 = lambda epoch: 0.99 **…
Penguin
  • 1,923
  • 3
  • 21
  • 51
3
votes
0 answers

How to tune learning rate with HParams Dashboard

In Tensorflow documentation, it is shown how to tune several hyperparameters but not the learning rate.I have searched how to tune learning rate using HParams dashboard but could not find much. The only example is another question on github but it…
Arwen
  • 168
  • 1
  • 13
3
votes
2 answers

How to select correct value of learning rate multiplier?

I want to manually select correct Learning rate in an image classification problem using Pytorch by running the model for few epochs. I have used LR scheduler to decay the learning rate and also have manipulated Learning rate in optimizer parameter…
Raam Mishra
  • 75
  • 5
3
votes
2 answers

How I can change learning rate of RLlib training agent in dynamic

I'm using ray RLlib library to train multi-agent Trainer on the 5-in-a-row game. This is zero-sum environment so I have a problem of agents behavior degeneration (always win for 1'st agent, 5 moves to win). I have an idea to change learning rate of…
Ярослав Линдер
  • 31
  • 1
  • 3
2
votes
1 answer

how MultiStepLR works in PyTorch

I'm new to PyTorch and am working on a toy example to understand how weight decay works in learning rate passed into the optimizer. When I use MultiStepLR , I was expecting to decrease the learning rate in given epoch numbers, however, it does not…
whitepanda
  • 471
  • 2
  • 12
2
votes
1 answer

Get current learning rate when using ReduceLROnPlateau

I am using ReduceLROnPlateau to modify the learning rate during training of a PyTorch mode. ReduceLROnPlateau does not inherit from LRScheduler and does not implement the get_last_lr method which is PyTorch's recommended way of getting the current…
Anil
  • 1,097
  • 7
  • 20
2
votes
1 answer

What exactly is meant by param_groups in pytorch?

I would like to update learning rates corresponding to each weight matrix and each bias in pytorch during training. The answers here and here and many other answers I found online talk about doing this using the model's param_groups which to the…
Toonia
  • 61
  • 1
  • 7
2
votes
0 answers

How to create a dynamic learning rate per neuron in PyTorch?

I know it's possible to have a learning rate per layer (link). I also found how to dynamically change the learning rate (changing it in the middle of training dynamically without a scheduler) (link). How can I create an optimizer that will have a…
Axo
  • 33
  • 5
2
votes
1 answer

How to implement randomised log space search of learning rate in PyTorch?

I am looking to fine tune a GNN and my supervisor suggested exploring different learning rates. I came across this tutorial video where he mentions that a randomised log space search of hyper parameters is typically done in practice. For sake of the…
Gonzo
  • 115
  • 5
1
2 3 4 5 6