Return value of highest index in numpy 3D array

Question

I have a 3D array in numpy that includes nans. I need to return the value with the greatest index position along the 0 axis. The answer would reduce to a 2D array.

There are a lot of questions about finding the index position of a maximum value along an axis (How to get the index of a maximum element in a numpy array along one axis), but that is different than what I need.

Example 3D array:

>>> import numpy as np
>>> foo = np.asarray([[[7,4,6],[4,2,11], [7,8,9], [4,8,2]],[[1,2,3],[np.nan,5,8], [np.nan,np.nan,10], [np.nan,np.nan,7]]])
>>> foo
array([[[  7.,   4.,   6.],
        [  4.,   2.,  11.],
        [  7.,   8.,   9.],
        [  4.,   8.,   2.]],

       [[  1.,   2.,   3.],
        [ nan,   5.,   8.],
        [ nan,  nan,  10.],
        [ nan,  nan,   7.]]])

I thought I was getting close using np.where but it returns all elements that are not nan. Not quite what I need because I want a (4,3) array.

>>> zoo = foo[np.where(~np.isnan(foo))]
>>> zoo
array([  7.,   4.,   6.,   4.,   2.,  11.,   7.,   8.,   9.,   4.,   8.,
     2.,   1.,   2.,   3.,   5.,   8.,  10.,   7.])

The answer I need is:

>>> ans = np.asarray([[1,2,3], [4,5,8], [7,8,10], [4,8,7]])
>>> ans
array([[ 1,  2,  3],
       [ 4,  5,  8],
       [ 7,  8, 10],
       [ 4,  8,  7]])

EDIT: I edited the foo example array to make the question more clear.

Thanks @Divakar but that isn't quite it. See my response to Edgar below. — kanerm, Dec 12 '18 at 19:48

Edgar Ramírez Mondragón · Answer 1 · 2018-12-12T20:24:11.067

1

You can use np.nanmax:

>>> np.nanmax(foo, axis=0)
array([[ 7.,  4.,  6.],
       [ 4.,  5., 11.],
       [ 7.,  8., 10.],
       [ 4.,  8.,  7.]])

The np.nanmax function returns the maximum of an array or maximum along an axis, ignoring any NaNs.

EDIT

As you rightly point out in your comment, you need the value at the maximum index and the code above doesn't return that.

Instead, you can use apply-along-axis:

>>> def highest_index(a):
...     return a[~np.isnan(a)][-1] # return non-nan value at highest index

>>> np.apply_along_axis(highest_index, 0, foo)
array([[ 1.  2.  3.]
       [ 4.  5.  8.]
       [ 7.  8. 10.]
       [ 4.  8.  7.]])

edited Dec 12 '18 at 20:24

answered Dec 12 '18 at 19:00

Edgar Ramírez Mondragón

2,912
3
19
34

Thanks Edgar, but that isn't what I need. Note the difference in your array and my ans. Your array returns the maximum value along the index (which at position 2,3 is 11) but I need the value at the maximum index (which at position 2,3 is 8). – kanerm Dec 12 '18 at 19:46
You are right! I edited my answer to produce the right result. – Edgar Ramírez Mondragón Dec 12 '18 at 20:18
Awesome! Thank you Edgar! – kanerm Dec 12 '18 at 20:30
Glad I could help! Please consider upvoting :) – Edgar Ramírez Mondragón Dec 12 '18 at 20:37
I sure did... apparently I don't have enough of a reputation yet for my vote to be on public display. Thanks for helping a new user. :) – kanerm Dec 12 '18 at 20:54
1

Just a note of warning, while `apply_along_axis` is convenient to use here, it isn't any faster than a more explicit iteration over the last 2 axes. – hpaulj Dec 12 '18 at 21:05
@hpaulj I wondered about iterating over the 0 axis but couldn't figure out how to do it. Are you willing to show an example answer? – kanerm Dec 12 '18 at 21:07
I was thinking terms applied this `highest_index(foo[:,i,j])` for all `i,j`. – hpaulj Dec 12 '18 at 21:13

B. M. · Answer 2 · 2018-12-13T07:22:19.097

0

A vectored solution, only with indices:

def last_non_nan(foo):
    i = np.isnan(foo)[::-1].argmin(0)
    j,k = np.indices(foo[0].shape)
    return foo[-i-1,j,k]

i contains the index of the first not nan number in the reversed 'line'. so -i-1 is its index in the direct line.

>>> last_non_nan(foo):
  [[  1.,   2.,   3.],
   [  4.,   5.,   8.],
   [  7.,   8.,  10.],
   [  4.,   8.,   7.]]

Faster than highest_index:

In [5]%timeit last_non_nan(foo)
133 µs ± 29.5 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [6]: %timeit np.apply_along_axis(highest_index,0,foo)
667 µs ± 90 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

up to 150x (40 ms vs 6 s) faster for a (10,400,400) array with 90 % nans.

It s essentially because last_non_nan just fetch the last non nan value in each line, when highest_index compute the index and fetch all non nan values.

edited Dec 13 '18 at 07:22

answered Dec 12 '18 at 21:55

B. M.

18,243
2
35
54

Thanks @B.M. I understand what `j,k` returns (the indices row wise and column wise). And it looks like returning `i ` was the step I was really missing. Could you walk me through what the line that returns `i` is doing? – kanerm Dec 12 '18 at 22:24
Also, I tried both your answer and @Edgar in my actual code problem (which is an array of shape `(13, 414, 394`). For some reason your answer works but Edgar's does not. I don't understand why... – kanerm Dec 12 '18 at 22:26
Thanks so much for the extra info @BM. I upvoted your answer, it just isn't public because of my new reputation. – kanerm Dec 13 '18 at 15:19

Return value of highest index in numpy 3D array

2 Answers2

EDIT