8

I am trying to implement leaky Relu, the problem is I have to do 4 for loops for a 4 dimensional array of input.

Is there a way that I can do leaky relu only using Numpy functions?

Liu Hantao
  • 620
  • 1
  • 9
  • 19

5 Answers5

12

Here are two approaches to implement leaky_relu:

import numpy as np                                                 

x = np.random.normal(size=[1, 5])

# first approach                           
leaky_way1 = np.where(x > 0, x, x * 0.01)                          

# second approach                                                                   
y1 = ((x > 0) * x)                                                 
y2 = ((x <= 0) * x * 0.01)                                         
leaky_way2 = y1 + y2
Amir
  • 16,067
  • 10
  • 80
  • 119
  • 1
    the second approach is a [bit faster](https://stackoverflow.com/a/49734841/6484358)! – Anu Mar 29 '19 at 00:42
2
import numpy as np




def leaky_relu(arr):
    alpha = 0.1
    
    return np.maximum(alpha*arr, arr)
vikanksh nath
  • 359
  • 3
  • 5
1

Going off the wikipedia entry for leaky relu, should be able to do this with a simple masking function.

output = np.where(arr > 0, arr, arr * 0.01)

Anywhere you are above 0, you keep the value, everywhere else, you replace it with arr * 0.01.

Lzkatz
  • 173
  • 8
0
def leaky_relu_forward(x, alpha):
  out = x                                                
  out[out <= 0]=out[out <= 0]* alpha
  return out
user17260419
  • 1
  • 1
0

Just to add, this approche :

def leaky_relu(x):
    return np.maximum(0.01*x, x)

is 2 time faster than this one :

leaky_way1 = np.where(x > 0, x, x * 0.01)
boblareleve
  • 1