0

I have saved my text vectors by using gensim library which consists of some negative numbers. will it effect the training? If not then why am i getting nan loss value first for discriminator and then for both discriminator and generator after certain steps of training?

Dhiraj Patnaik
  • 31
  • 1
  • 8
  • here a detailed answer of possible reasons of getting NAN loss https://stackoverflow.com/questions/40050397/deep-learning-nan-loss-reasons – Aziz Sep 06 '18 at 20:29

1 Answers1

4

There are several reasons for a NaN loss and why models diverge. Most common ones I've seen are:

  • Your learning rate is too high. If this is the case, the loss increases and then diverges to infinity.
  • You are getting a division by zero error. If this is the case, you can add a small number like 1e-8 to your output probability.
  • You have bad inputs. If this is the case, make sure that you do not feed your model with NaNs. i.e. use assert not np.any(np.isnan(x)) on the input data.
  • Your labels are not in the same domain of your objective function. If this is the case, check the range of your labels and make sure they match.

If none of the above helps, try to check the activation function, the optimizer, the loss function, the size and the shape of the network.

Finally, though less likely, there might be a bug with the framework you are using. Check the repo of the framework if there are others having the same issue.

Semih Yagcioglu
  • 4,011
  • 1
  • 26
  • 43
  • Ok that i will try but first i need a clarification. I have dataset of colour images in which some are in form of RGB and some are in form of RGBA. How should i feed them into my model? – Dhiraj Patnaik Sep 06 '18 at 21:44
  • And furthermore i am getting nan loss value first for generator. – Dhiraj Patnaik Sep 06 '18 at 22:25
  • You need to standardize your input and feed your network with the same type of data. Preprocessing is very important while developing ML models. – Semih Yagcioglu Sep 07 '18 at 03:58
  • I am doing that and i came to know that learning rate was a bit high so by lowering it i am able to run smoothly. thanks – Dhiraj Patnaik Sep 07 '18 at 07:33
  • Glad to hear that. If this solves your problem, mark it as solved so that others can benefit from the solution. – Semih Yagcioglu Sep 07 '18 at 09:00
  • ok will do that but i needed to know how should i normalise the img numpy array between -1 and 1 for training stability? please let me know – Dhiraj Patnaik Sep 07 '18 at 13:10
  • Here's a link on how to scale your data: https://stackoverflow.com/questions/1735025/how-to-normalize-a-numpy-array-to-within-a-certain-range – Semih Yagcioglu Sep 07 '18 at 13:27