4

I stumpled upon some behaviour of the np.array function I do not understand. This code works the way I expect it:

arr1 = np.zeros((3,2))
arr2 = np.zeros((2,2))

np.array([arr1,arr2])

array([ array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.]]),
       array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.],
       [ 0.,  0.,  0.]])], dtype=object)

But this code gives me an error:

arr1 = np.zeros((2,3))
arr2 = np.zeros((2,2))

np.array([arr1,arr2])

ValueError: could not broadcast input array from shape (2,3) into shape (2)

Why does it make a difference, whether the size of the first dimension matches? And how can I force the function to behave as it does in the first example?

Eric
  • 95,302
  • 53
  • 242
  • 374
sietschie
  • 7,425
  • 3
  • 33
  • 54
  • What is your expected shape of the matrix in the second example then? – kennytm Feb 28 '17 at 14:30
  • I would expect the same shape as the first one: `(2,)` – sietschie Feb 28 '17 at 14:31
  • You can use the following hack: `newarr=np.array([None, arr2, arr3, etc.]);newarr[0]=arr1` – Paul Panzer Feb 28 '17 at 14:40
  • Maybe a bug? http://stackoverflow.com/questions/32909619/numpy-array-1-9-2-getting-valueerror-could-not-broadcast-input-array-from-shape – kennytm Feb 28 '17 at 14:42
  • Does `np.array([arr1,arr2], dtype=object)` work? Edit: no – Eric Feb 28 '17 at 14:56
  • `np.array([arr1.reshape(1,2,3), arr2])` should give the shape you're after. I guess the `ValueError` gets emitted if their first dimension, `a.shape[0]` are overlapping and also whose overall shape aren't the same throughout. With Broadcasting, it would be `np.array([arr1[None,:],arr2])` – Nickil Maveli Feb 28 '17 at 15:04

1 Answers1

2

As noted in numpy array 1.9.2 getting ValueError: could not broadcast input array from shape (4,2) into shape (4)

there have been changes in how object arrays are created. Where possible np.array tries to create multidimensional array. Creating an object array has been the fallback choice, to be used when the inputs don't match in shape. And even then it can be unpredictable.

The surest way is to make an empty object array of the right size and insert the objects

In [242]: a=np.zeros((2,3));b=np.ones((2,2))

In [243]: arr=np.zeros((2,), object)
In [244]: arr[0]=a; arr[1]=b

In [245]: arr
Out[245]: 
array([array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.]]),
       array([[ 1.,  1.],
       [ 1.,  1.]])], dtype=object)
In [246]: arr[:]=[a,b]           # also works

You've already seen that np.array([a.T, b]) works. If I turn the arrays into nested lists I get a 2d array of flat lists; 2 and 3 element lists are the lowest level of incompatible objects.

In [250]: np.array([a.tolist(),b.tolist()])
Out[250]: 
array([[[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]],
       [[1.0, 1.0], [1.0, 1.0]]], dtype=object)

In [251]: _.shape
Out[251]: (2, 2)

np.array is compiled, so it would take some digging to decode its logic. My earlier digging indicates that the made some changes to speed up the creation of an array from arrays (more like concatenate). This problem may be a side effect of that change.

The take home message is: if you want to create an object array of arrays, don't count on np.array to do the job right. It's designed primarily to make multidimensional arrays of scalars.

For example if the arrays match in size:

In [252]: np.array([a,a]).shape
Out[252]: (2, 2, 3)

I have to use arr[...]=[a,a] if I want a 2 element object array.

Community
  • 1
  • 1
hpaulj
  • 221,503
  • 14
  • 230
  • 353