Curious numpy advanced indexing selection case

Question

Here is a Python program that uses advanced indexing to slice a 3D array in three slightly different ways to arrive at the same result. a and b show in the same expected result. However c shows an error message "IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (3,) (2,) (1,)". The only difference between a, b, and c is the slightly different way of selecting the range of second indices. Why is c erroring out:

import numpy as np

y = np.arange(0,18)
y1 = y.reshape((3,2,3))

print("y1 is \n{}\n".format(y1))

#a is ok
a = y1[[0,1,2],:,[False,False,True]]
print("a is \n{}\n".format(a))

#b is ok, same as a
b = y1[[0,1,2],0:2,[False,False,True]]
print("b is \n{}\n".format(b))

#below errors out
c = y1[[0,1,2],[0,1],[False,False,True]]
print("c is \n{}\n".format(c))

The last case needs `ix_` broadcasting. One slice in the middle can mess with the shape order. It's a mixed advanced/basic indexing case. Also look at `[:,:,[2]]` — hpaulj, Oct 02 '17 at 08:19
numpy changes its indexing behavior depending on if it encounters a list or a slice object. You can read about it in fancy indexing section in the docs. — percusse, Oct 02 '17 at 08:57

score 0 · Answer 1 · answered Oct 02 '17 at 08:04

0

Using False as index 0 and True as index 1 is a little bit confusing.

a = y1[[0,1,2],:,[False,False,True]]

or better

a = y1[[0,1,2],:,[0,0,1]]

means, take the first elements of each list as index, then the second elements, ...

a = [y1[0, :, 0], y1[1, :, 0], y1[2, :, 1]]

In the case of c, the middle list is to short, the third element is missing.

Another example, to make it clear

d = y1[[0,2,0],[0,1,0],[2,2,1]]

is

d = [y1[0, 0, 2], y1[2, 1, 2], y1[0, 0, 1]]

answered Oct 02 '17 at 08:04

Daniel

42,087
4
55
81

In this case the boolean index is the same as `[2]`, not [0,0,1]. It picks 1 column on the last dimension. – hpaulj Oct 02 '17 at 08:14
@hpaulj: In this case, it is exactly, what I've written. Test it yourself. – Daniel Oct 02 '17 at 09:13
With the boolean indexing I get the equivalent of `y1[:,:,2]`. – hpaulj Oct 02 '17 at 20:46

hpaulj · Answer 2 · 2017-10-02T17:07:08.547

The issues here are:

how a boolean indexing works
how indexing lists (arrays) broadcast with each other
how mixed advanced/basic indexing works.

In [286]: y1 = np.arange(18).reshape(3,2,3)
In [287]: y1
Out[287]: 
array([[[ 0,  1,  2],
        [ 3,  4,  5]],

       [[ 6,  7,  8],
        [ 9, 10, 11]],

       [[12, 13, 14],
        [15, 16, 17]]])
In [288]: y1[[0,1,2],:,[False, False, True]]
Out[288]: 
array([[ 2,  5],
       [ 8, 11],
       [14, 17]])
In [289]: y1[[0,1,2],0:2,[False, False, True]]
Out[289]: 
array([[ 2,  5],
       [ 8, 11],
       [14, 17]])
In [290]: y1[[0,1,2],[0,1],[False, False, True]]
....
IndexError: shape mismatch: indexing arrays could not be 
broadcast together with shapes (3,) (2,) (1,)

So the first 2 produce a (3,2) result, and the last an error. Note that the last has converted the [False,False,True] into a (1,) indexing array.

The first is the same as this simpler indexing:

In [294]: y1[:,:,2]
Out[294]: 
array([[ 2,  5],
       [ 8, 11],
       [14, 17]])

If instead I use the boolean, it's as though I used y1[:,:,[2]]]:

In [291]: y1[:,:,[False,False,True]]
Out[291]: 
array([[[ 2],
        [ 5]],

       [[ 8],
        [11]],

       [[14],
        [17]]])

A (3,2,1) array.

y1[np.ix_([0,1,2], [0,1], [2])] produces the same thing, but with advanced indexing. The boolean list works in the same way. ix_ reshapes the 3 arrays so they can broadcast against each other

In [299]: np.ix_([0,1,2],[0,1],[2])
Out[299]: 
(array([[[0]],

        [[1]],

        [[2]]]), array([[[0],
         [1]]]), array([[[2]]]))

This creates a (3,1,1), (1,2,1), and (1,1,1) which broadcast to (3,2,1).

y1[[0,1,2], :, [2]] also uses broadcasting, a (3,) with a (1,) to produce a (3,). The (2,) middle dimension is tacked on to the end. This an example of mixed basic/advanced indexing. In some cases this is counter intuitive. For example, y1[[2], :, [0,1,2]] produces a (3,2), whereas users often expect (2,3).

Python multi-dimensional notation transpose automatically

The boolean indexing is not the same as a 0/1 indexing.

In [297]: y1[[0,1,2], :, [0,0,1]]
Out[297]: 
array([[ 0,  3],
       [ 6,  9],
       [13, 16]])

This selects y1[0,:,0], y1[1,:,0], and y1[2,:,1]. It pairs up the elements of the 2 lists. It broadcasts a (3,) with a (3,). Again the middle slice is added at the end.

Note that the boolean mask could have problems if it was [True, False, True], which is the equivalent of [0,2], a size (2,) indexing. That does not broadcast with (3,).

In [307]: y1[[0,1,2],:,[True, False, True]]
IndexError: shape mismatch: indexing arrays could not be broadcast together with shapes (3,) (2,)

Test your understanding with y1[[[0],[1],[2]],:,[True, False, True]] (3,2,2) result, and y1[[[0],[1],[2]],[0,1],[True, False, True]] (3,2).

`For example, y1[[2], :, [0,1,2]] produces a (3,2), whereas users often expect (2,3).`: In my current perspective, during broadcasting it's syncing (1,),(),(3,) -> (3,) and finally the length of the second dimension increases to `2`. So indeed I got `(2,3)`. What's the problem of this perspective? — NeoZoom.lua, Mar 16 '22 at 18:08
What a depressing case, I don't know how to understand it. Is this really a bug or I'm missing something? — NeoZoom.lua, Mar 16 '22 at 18:47
@Rainning, only the [2] and [0,1,2] are used in the `broadcasting`. `y1[[1], [[0],[1]], [0, 1, 2]]` broadcasts all three, giving the expected (2,3). I have not explored the compiled code, so can only cite the docs, and illustrate cases. — hpaulj, Mar 16 '22 at 20:00

Curious numpy advanced indexing selection case

2 Answers2