1

I'm implementing a NN library and I wrote the softmax activation function like such:

    def function(self, z):
        maxValue = np.max(z)
        e_x = np.exp(z - maxValue)
        returnValue = e_x / e_x.sum(axis=0)
        return returnValue

as you can see, it's a regular softmax. I thought using maxValue would be enough to make the values in the right range, but they still output values which are so small, in the division, python can't handle them so it converts them to NaN. That messes up my program and the model doesn't learn because the output is NaN. How do you suggest fixing this? I tried using bigFloat but it seems it isn't compatible with numpy arrays.

desertnaut
  • 57,590
  • 26
  • 140
  • 166
  • What is the value of `z.shape`? This implementation works only for 1d arrays. – V. Ayrat Jun 21 '20 at 22:10
  • Hey @V. Ayrat I would like to reference you to this post - https://stackoverflow.com/questions/34968722/how-to-implement-the-softmax-function-in-python –  Jun 22 '20 at 05:03
  • You don't answer the question about `z.shape`. You get this error when you call your function for 2d array, am I right? – V. Ayrat Jun 22 '20 at 16:11
  • @V.Ayrat This is a runtime warning not an error, something along the lines of underflow detected... it does run with 2d matrices, even 5d matrices, the problem is the values it outputs, not the way of calculating. Well at least, as far as I've seen according to the post I sent. One of the answers clearly states that to apply to multi-dimensional matrices, we need the .sum(axis=0) line. –  Jun 22 '20 at 20:16
  • I don't ask it out of curiosity. If this happens for 2d arrays you should just change np.max(z) to np.max(z, axis=0). But it is not wise to write detailed explanation why this underflow happens just to find out what this happens also for 1d arrays where my explanation will not work and probably something else happens. – V. Ayrat Jun 22 '20 at 21:27

0 Answers0