Questions tagged [gradient-exploding]
5 questions
1
vote
0 answers
Tensorflow, gradients become NAN even when I clip it
It seems like I have an exploding gradient issue during the training of my reinforcement learning policy.
However, I'm using a gradient clipping by norm with 0.2 as the clipping factor.
I've check both my inputs and my loss and none of them are NAN.…
Aydin Abiar
- 334
- 3
- 11
1
vote
1 answer
LSTM network loss is nan for batch size bigger than one
I am trying to analyse EEG data using LSTM network, I split the data into 4 seconds segment which resulted in around 17000 data samples. To that end, I build the following network bellow:
def load_model():
model = Sequential()
…
Khaled Bagh
- 61
- 4
0
votes
1 answer
An example of how pytorch clip_grad_norm_ works
I'd like a simple example to illustrate how gradient clipping via clip_grad_norm_ works. From this post, I found that if the norm of a gradient is greater than a threshold, then it simply takes the unit vector of the gradient and multiplies it with…
max_max_mir
- 1,494
- 3
- 20
- 36
0
votes
0 answers
PyTorch simple ConvNet diverge so easly
So I'm studiying pytorch coming from a background with tensorflow.
I'm trying to replicate a simple convnet, that I've developed with success in tensorflow, to classify cat vs dogs images.
In pytorch I see some strange behaviors:
Using a Learning…
Levenlol
- 305
- 5
- 17
0
votes
1 answer
In Keras, using SGD, why model.fit() trains smoothly, but step wise training method gives exploding gradient and loss
Because this exploding gradients and exploding loss happens when the network is huge, so I don't bother post the entire network here. But I've tried my best, the past two weeks, I dig down into the very details of the source code to monitor some…
Jason
- 3,166
- 3
- 20
- 37