2

I declared ReLU function like this:

def relu(x):
    return (x if x > 0 else 0)

and an ValueError has occured and its traceback message is

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

But if I change ReLU function with numpy, it works:

def relu_np(x):
    return np.maximum(0, x)

Why this function(relu(x)) doesn't work? I cannot understand it...

================================

Used code:

>>> x = np.arange(-5.0, 5.0, 0.1)
>>> y = relu(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "filename", line, in relu
    return (x if x > 0 else 0)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
i.meant.to.be
  • 31
  • 4
  • Where and how are you calling this function? – Tobi208 Jan 26 '22 at 08:57
  • I used numpy.arange, and will add a code on this question right now – i.meant.to.be Jan 26 '22 at 09:01
  • your ternary operation `(x if x > 0 else 0)` is trying to do an if, else based on `x>0` which expects a single value and not an array. This is the reason for the error. However, for numpy you are using a vectorized function `np.maximum` which appies independently to each element of the array. Check details in my answer below. – Akshay Sehgal Jan 26 '22 at 09:39

3 Answers3

2

TLDR; Your first function is not using vectorized methods which means it expects a single float/int value as input, while your second function takes advantage of Numpy's vectorization.

Vectorization in NumPy

Your second function uses numpy functions which are vectorized and run on each individual element of the array.

import numpy as np

arr = np.arange(-5.0, 5.0, 0.5)

def relu_np(x):
    return np.maximum(0, x)

relu_np(arr)

# array([0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. ,
#        1.5, 2. , 2.5, 3. , 3.5, 4. , 4.5])

Your second function however uses a ternary operator (x if x > 0 else 0) which expects a single value input and outputs a single value. This is why when you pass a single element, it would work, but on passing an array it fails to run the function on each element independently.

def relu(x):
    return (x if x > 0 else 0)

relu(-8)

## 0

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Note: The reason for this error is due to the ternary operator you are using (x if x > 0 else 0). The condition x>0 can only take the value True or False for a given integer/float value. However, when you pass an array, you will need to use something like any() or all() to aggregate that list of boolean values to a single one, before you can apply your if, else clause.

Solutions -

There are a few ways you can make this work -

1. Using np.vectorize (not recommended, lower performance than pure numpy approach)

import numpy as np

arr = np.arange(-5.0, 5.0, 0.5)

def relu(x):
    return (x if x > 0.0 else 0.0)

relu_vec = np.vectorize(relu)
relu_vec(arr)

# array([0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. ,
#        1.5, 2. , 2.5, 3. , 3.5, 4. , 4.5])

2. Iteration over the array with list comprehension

import numpy as np

arr = np.arange(-5.0, 5.0, 0.5)

def relu(x):
    return (x if x > 0 else 0)

arr = np.array(arr)

np.array([relu(i) for i in arr])

# array([0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0. , 0.5, 1. ,
#        1.5, 2. , 2.5, 3. , 3.5, 4. , 4.5])
Akshay Sehgal
  • 18,741
  • 3
  • 21
  • 51
  • It seems to me that your solution 2 would be the one with lowest performance. In fact, here is what i get : pure numpy : 13.9 microseconds, vectorise : 1.5 ms, iteration : 5.2 ms (need a fairly large array to illustrate the fact that iteration is slower than the vectorized form) – Colin Bernet Jan 26 '22 at 09:58
  • 1
    Apologies but I wasn't comparing the 2 approaches, just added the last approach later. The low performance is w.r.t the pure numpy approach vs vectorized approach. Let me edit that to clarify. – Akshay Sehgal Jan 26 '22 at 13:18
1

Keep in mind that x > 0 is an array of booleans, a mask if you like:

array([False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True])

So it does not make sense to do if x>0, since x contains several elements, which can be True or False. This is the source of your error.

Your second implementation of numpy is good ! Another implementation (maybe more clear?) might be:

def relu(x):
  return x * (x > 0)

In this implementation, we do an elementwise multiplication of x, which is a range of values along the x axis, by 0 if the element of x is below 0, and 1 if the element is above.

Colin Bernet
  • 1,354
  • 9
  • 12
1

Disclaimer: please someone correct me if I'm wrong, I'm not 100% sure about how numpy does things.

Your function relu expects a single numerical value and compares it to 0 and returns whatever is larger. x if x > 0 else 0 would be equal to max(x, 0) where max is a builtin Python function.

relu_np on the other hand uses the numpy function maximum which accepts 2 numbers or arrays or iterables. That means you can pass your numpy array x and it applies the maximum function to every single item automatically. I believe this is called 'vectorized'.

To make the relu function you have work the way it is, you need to call it differently. You'd have to manually apply your function to every element. You could do something like y = np.array(list(map(relu, x))).

Tobi208
  • 1,306
  • 10
  • 17