How to implement the derivative of Leaky Relu in python?

Question

How would I implement the derivative of Leaky ReLU in Python without using Tensorflow?

Is there a better way than this? I want the function to return a numpy array

def dlrelu(x, alpha=.01):
     # return alpha if x < 0 else 1

     return np.array ([1 if i >= 0 else alpha for i in x])

Thanks in advance for the help

What is `x`? Is it an array already? – roganjosh Jan 04 '18 at 20:14 — roganjosh, Jan 04 '18 at 20:14
Where is TensorFlow in your code? – grovina Jan 04 '18 at 20:35 — grovina, Jan 04 '18 at 20:35
@grovina "_without_ using Tensorflow" – roganjosh Jan 04 '18 at 20:36 — roganjosh, Jan 04 '18 at 20:36
@roganjosh Oops! Sorry, my bad! – grovina Jan 04 '18 at 20:37 — grovina, Jan 04 '18 at 20:37
Yes, x is n numpy array! @roganjosh – Lécio Bourbon Jan 05 '18 at 00:38 — Lécio Bourbon, Jan 05 '18 at 00:38

Gerges · Accepted Answer · 2018-01-04T20:32:40.763

13

The method you use works, but strictly speaking you are computing the derivative with respect to the loss, or lower layer, so it might be wise to also pass the value from lower layer to compute the derivative (dl/dx).

Anyway, you can avoid using the loop which is more efficient for large x. This is one way to do it:

def dlrelu(x, alpha=0.01):
  dx = np.ones_like(x)
  dx[x < 0] = alpha
  return dx

If you passed the error from lower layer, it looks like this:

def dlrelu(dl, x, alpha=0.01):
  """ dl and x have same shape. """
  dx = np.ones_like(x)
  dx[x < 0] = alpha
  return dx*dl

edited Jan 04 '18 at 20:32

answered Jan 04 '18 at 20:20

Gerges

6,269
2
22
44

Thanks a lot. I´m doing this multiplication "dx*dl" outside of the defined function... – Lécio Bourbon Jan 05 '18 at 01:00

score 1 · Answer 2 · answered Apr 09 '22 at 11:31

1

A simple convenience function:

def leakyReLU_deriv(x, alpha=0.01):
    return np.where(x>0, 1, alpha)

answered Apr 09 '22 at 11:31

Anil Myne

79
3

How to implement the derivative of Leaky Relu in python?

2 Answers2

Linked