Numpy broadcast shape error: convert list( lists(arrays) ) to array

Question

I have a list of lists like list_ABC = [[A,B,C], [A,B,C], ...]
with 2D ndarrays A (2x2), B (2x3) and C (2x3).

Now, I'd like to convert the main list to a numpy array:

np.array(list_ABC)

However, I get the following error:

ValueError: could not broadcast input array from shape (2,2) into shape (2)

I need this conversion because I'd like to get

A_matrices = np.array(list_ABC)[:, 0]
B_matrices = np.array(list_ABC)[:, 1]

Such that I can finally obtain a ndarray containing all A-arrays (array(A,A,A,...)).

Unfortunately I can't get a clue from the value error message. Interestingly, if I only transpose matrix C with C.T (making it a 3x2 matrix) no error is thrown.

Now, I could solve the problem by creating a list_A, list_B, list_C beforehand (and not list_ABC), but this doesn't feel as simple (constructing and appending to each list_A/B/C requires a few more lines of code). Similarly I could use other methods (e.g. using a dict with A,B,C keys containing a list of all A/B/C matrices), but nothing feels so simple like this solution.

A working example which throws the error:

import numpy as np
list = [[np.array([[ 476.,  667.], [ 474.,  502.]]), np.array([[ 343.,  351.,  449.], [ 352.,  332.,  292.]]), np.array([[ 328.,  328.,  294.], [ 367.,  355.,  447.]])], [np.array([[ 497.,  546.], [ 456.,  517.]]), np.array([[ 361.,  342.,  340.], [ 341.,  304.,  328.]]), np.array([[ 347.,  313.,  293.], [ 355.,  333.,  375.]])]]
np.array(list)

Thanks a lot!

I think it's because some of the arrays have two elements and some three! — Mahdi, May 19 '17 at 13:49
Yes if all of your arrays in the example are 2x3, then it works. — TMrtSmith, May 19 '17 at 13:52
Unfortunately the arrays need to be of these dimensions. And as written, somehow it works if I only transpose C? — 0vbb, May 19 '17 at 13:59
Seems like it's something to do with how np.array is trying to make the new array of arrays. If you play around with C.T, or making them all 2x3 or all 3x3, then look at the shape, you either get a 2D array of arrays, or a 4D array of arrays. I'm guessing that this is why the error is thrown up when the columns of A don't match the columns of C — TMrtSmith, May 19 '17 at 14:30
Constructing an array from other arrays creates a higher dimensional array if shapes match, object dtype array if they don't, or throws this error for certain types of mismatch. — hpaulj, May 19 '17 at 15:41

score 2 · Accepted Answer · edited May 23 '17 at 12:10

When constructing an array from arrays, np.array function can do 3 things:

if all subarrays have the same shape it will make a higher dimensional array
if the subarrays differ in shape it may construct an object dtype array. This is like a list, or nested list, but with the ability to index and reshape like arrays
raise an error. This seems to happen most when rows match but columns don't. It may be detecting the shape mismatch too late to fall back on the object dtype solution.

More on this at:

How to keep numpy from broadcasting when creating an object array of different shaped arrays

To get array(A,A,A,...) I would suggest use list comprehension. Constructing object dtype arrays is too tricky.

The reliable way of creating an object array is to initialize an empty one and fill it:

In [116]: arr = np.empty((2,3), dtype=object)
In [117]: arr[...] = alist
In [118]: arr
Out[118]: 
array([[array([[ 476.,  667.],
       [ 474.,  502.]]),
        array([[ 343.,  351.,  449.],
       [ 352.,  332.,  292.]]),
        array([[ 328.,  328.,  294.],
       [ 367.,  355.,  447.]])],
       [array([[ 497.,  546.],
       [ 456.,  517.]]),
        array([[ 361.,  342.,  340.],
       [ 341.,  304.,  328.]]),
        array([[ 347.,  313.,  293.],
       [ 355.,  333.,  375.]])]], dtype=object)

Now I can select the 'A' elements:

In [119]: arr[:,0]
Out[119]: 
array([array([[ 476.,  667.],
       [ 474.,  502.]]),
       array([[ 497.,  546.],
       [ 456.,  517.]])], dtype=object)

but that's an object array, and wrapping again in np.array doesn't change that:

In [120]: np.array(arr[:,0])
Out[120]: 
array([array([[ 476.,  667.],
       [ 474.,  502.]]),
       array([[ 497.,  546.],
       [ 456.,  517.]])], dtype=object)

but they can be concatenated on several different axes.

In [121]: np.stack(arr[:,0])
Out[121]: 
array([[[ 476.,  667.],
        [ 474.,  502.]],

       [[ 497.,  546.],
        [ 456.,  517.]]])

But I can get the same thing without the object array step

In [123]: np.stack([a[0] for a in alist])
Out[123]: 
array([[[ 476.,  667.],
        [ 474.,  502.]],

       [[ 497.,  546.],
        [ 456.,  517.]]])

Thanks a lot for the detailed explanation! The list comprehension method is very slim and easy :) — 0vbb, May 22 '17 at 10:21

Numpy broadcast shape error: convert list( lists(arrays) ) to array

1 Answers1