1

Consider that you are using a Pytorch optimizer such as torch.optim.Adam(model_parameters).

So in your training loop you will have something like: 

optimizer = torch.optim.Adam(model_parameters)
# put the training loop here

loss.backward()
optimizer.step()
optimizer.zero()

Is there a way to monitor what steps are taking your optimizer ? To make sure that you are not on a flat area and thus taking no steps since the gradient are null. Maybe checking the learning rate would be a solution ?

White
  • 305
  • 2
  • 11
  • 1
    You can just make sure your model hasn't converged by monitoring that your loss is decreasing every few iterations – Mike May 12 '20 at 13:41

1 Answers1

2

Answering to myself here.

The best practice would be (in PyTorch) to check the gradient of the leaf tensors. If the gradient is None but is_leaf attribute is set to True, something is obviously buggy.

torch.nn.Parameters('insert your tensor here') is especially confusing to this regard. As the Tensor needs to be defined as a torch.nn.Parameters to be succesfully updated. I would advise not to use tensor.requires_grad_(True) as is confuses torch. Only set your parameters as above.

White
  • 305
  • 2
  • 11