Implement Relu derivative in python numpy

Question

I'm trying to implement a function that computes the Relu derivative for each element in a matrix, and then return the result in a matrix. I'm using Python and Numpy.

Based on other Cross Validation posts, the Relu derivative for x is 1 when x > 0, 0 when x < 0, undefined or 0 when x == 0

Currently, I have the following code so far:

def reluDerivative(self, x):
    return np.array([self.reluDerivativeSingleElement(xi) for xi in x])

def reluDerivativeSingleElement(self, xi):
    if xi > 0:
        return 1
    elif xi <= 0:
        return 0

Unfortunately, xi is an array because x is an matrix. reluDerivativeSingleElement function doesn't work on array. So I'm wondering is there a way to map values in a matrix to another matrix using numpy, like the exp function in numpy?

Thanks a lot in advance.

[np.heaviside](https://docs.scipy.org/doc/numpy/reference/generated/numpy.heaviside.html) — Neil G, Sep 25 '17 at 17:40

Jakub Bartczuk · Answer 1 · 2018-01-01T12:46:17.383

27

That's an exercise in vectorization.

This code

if x > 0:
  y = 1
elif xi <= 0:
  y = 0

Can be reformulated into

y = (x > 0) * 1

This is something that will work for numpy arrays, since boolean expressions involving them are turned into arrays of values of these expressions for elements in said array.

edited Jan 01 '18 at 12:46

answered Sep 25 '17 at 17:55

Jakub Bartczuk

2,317
1
20
27

I put this code in my reluDerivativeSingleElement function. It still says it cannot work with array. Maybe I interpret your answer wrongly? – Bon Sep 26 '17 at 01:08
It doesn't make much sense - this code is supposed to work on whole array – Jakub Bartczuk Sep 26 '17 at 09:08
1

Why not simply `y = (x > 0)*1.0` Seems to produce the same results as `y = (x > 0) * 1 + (x <= 0) * 0` – Bill Jan 01 '18 at 02:25

score 23 · Accepted Answer · answered Sep 25 '17 at 18:34

23

I guess this is what you are looking for:

>>> def reluDerivative(x):
...     x[x<=0] = 0
...     x[x>0] = 1
...     return x

>>> z = np.random.uniform(-1, 1, (3,3))
>>> z
array([[ 0.41287266, -0.73082379,  0.78215209],
       [ 0.76983443,  0.46052273,  0.4283139 ],
       [-0.18905708,  0.57197116,  0.53226954]])
>>> reluDerivative(z)
array([[ 1.,  0.,  1.],
       [ 1.,  1.,  1.],
       [ 0.,  1.,  1.]])

answered Sep 25 '17 at 18:34

Irshad Bhat

8,479
1
26
36

Thanks, I made copy of x so that the original x is not modified. – Bon Sep 26 '17 at 01:07
@Bon, why do you think original x will be modified? I suppose, `x` is local variable for `reluDerivative` function, it shouldn't affect `x` outside that scope, isn't it? – Rishabh Agrahari Dec 26 '17 at 19:25
3

@RishabhAgrahari `x` would be a numpy array that is passed by reference so modifying `x` within `reluDerivative` would modify the original `x` that is passed into the function – Ron7 Feb 19 '18 at 05:15
@Bon Thanks. But how do you know `x` is passed by reference, this doesn't happen generally, right? – Rishabh Agrahari Feb 19 '18 at 05:25
1

Just want to note that this is technically just one of many programmatic solution since the derivative of ReLU is undefined at 0. So instead of `x[x<=0] = 0` other common implementations that work in practice are `x[x<0] = 0` & `x[x==0] = 1` and `x[x<0] = 0` & `x[x==0] = 0.5` – Jun 02 '18 at 18:56
def dRelu(z): return np.where(z <= 0, 0, 1) – drbombe May 15 '20 at 13:17

score 19 · Answer 3 · answered Nov 19 '17 at 17:50

19

Basic function to return derivative of relu could be summarized as follows:

f'(x) = x > 0

So, with numpy that would be:

def relu_derivative(z):
    return np.greater(z, 0).astype(int)

answered Nov 19 '17 at 17:50

milosdju

783
12
27

1

This seems to be the fastest implementation provided in this thread. – Skillmon May 06 '18 at 19:52

score 10 · Answer 4 · answered Oct 31 '18 at 14:16

10

def dRelu(z):
    return np.where(z <= 0, 0, 1)

Here z is a ndarray in my case.

answered Oct 31 '18 at 14:16

Krishna Mishra

101
1
6

user3503711 · Answer 5 · 2023-08-22T14:21:35.727

5

def reluDerivative(self, x): 
    return 1 * (x > 0)

Edit: OP had self in the function parameter as it was part of a class.

edited Aug 22 '23 at 14:21

answered May 07 '19 at 16:45

user3503711

1,623
1
21
32

1

@OlivierD'Ancona Why? Then for ```x=0```, the function will return 1. – user3503711 May 03 '22 at 19:22
You're right because the derivative is not defined in x = 0 with lim x->0- we have 0 and with lim x->0+ we have 1. Therefore the derivative isn't continuous. Thank you – Olivier D'Ancona May 04 '22 at 19:54

score 0 · Answer 6 · answered Sep 25 '17 at 17:43

You are on a good track: thinking on vectorized operation. Where we define a function, and we apply this function to a matrix, instead of writing a for loop.

This threads answers your question, where it replace all the elements satisfy the condition. You can modify it into ReLU derivative.

https://stackoverflow.com/questions/19766757/replacing-numpy-elements-if-condition-is-met

In addition, python supports functional programming very well, try to use lambda function.

https://www.python-course.eu/lambda.php

score 0 · Answer 7 · answered Dec 22 '17 at 05:52

0

This works:

def dReLU(x):
    return 1. * (x > 0)

answered Dec 22 '17 at 05:52

Shital Shah

63,284
17
238
185

score 0 · Answer 8 · answered Dec 26 '17 at 19:57

0

As mentioned by Neil in the comments, you can use heaviside function of numpy.

def reluDerivative(self, x):
    return np.heaviside(x, 0)

answered Dec 26 '17 at 19:57

Rishabh Agrahari

3,447
2
21
22

score 0 · Answer 9 · answered Mar 31 '19 at 18:34

0

If you want to use pure Python:

def relu_derivative(x):
    return max(sign(x), 0)

answered Mar 31 '19 at 18:34

Kirill Dolmatov

327
5
11

Olivier D'Ancona · Answer 10 · 2022-05-04T19:56:11.190

0

If you want it with the derivative you can use:

def relu(neta):
    relu = neta * (neta > 0)
    d_relu = (neta > 0)
    return relu, d_relu

edited May 04 '22 at 19:56

answered Mar 11 '22 at 13:20

Olivier D'Ancona

779
2
14
30

score -1 · Answer 11 · answered Dec 14 '17 at 16:48

-1

When x is larger than 0, the slope is 1. When x is smaller than or equal to 0, the slope is 0.

if (x > 0):
    return 1
if (x <= 0):
    return 0

This can be written more compact:

return 1 * (x > 0)

answered Dec 14 '17 at 16:48

marnix

60
8

This doesn't answer the question, and although more compact is less readable. – bphi Dec 14 '17 at 17:08

Implement Relu derivative in python numpy

11 Answers11

Linked