Reshaping numpy array to be similar to different array with arbitrarily nested subarrays

Question

I have two arrays a and b. If I run a.shape I get (10,) and if I run (b.shape) I get (10,) however each of those values are arbitrarily nested arrays which are not equal in shape to the others in itself or to the corresponding value in the other array. For example a[0].shape returns (122,) and b[0].shape returns (3900,) and b[5].shape returns (5200,64) so they aren't even consistent within the same array.

I know that they have the same total number of elements using the recursive solution found at Iterating through a multidimensional array in Python (no standard numpy functions seemed to be able to drill as deeply as necessary into the arrays):

def iterThrough(lists):
  if not hasattr(lists[0], '__iter__'):
    for val in lists:
      yield val
  else:
    for l in lists:
      for val in iterThrough(l):
        yield val
            si = 0

for value in iterThrough(a): #and b
    si += 1

Both return 3066752. I understand that this is quite messy, but I need it to be in that shape to compute a mathematical function later on and I'm guessing it's easier to to do this than to rewrite that equation to reflect a nicely formatted array.

How can I make array a exactly identical in all its nested shapes to array b?

If you have this weirdly shaped data, I don't understand why you are using Numpy or what benefit you hope to gain from it. Anyway, your question as asked makes no sense: what do you actually want to do with the values? What is the rule that tells you where any given element from the original array should go in the reshaped array? — Karl Knechtel, Nov 27 '20 at 19:44
i don't understand what your asking, if you "need it to be in that shape" but want the two arrays "exactly identical". Can you post an expected output and the current shapes — Kenan, Nov 27 '20 at 19:48
@Kenan It doesn't need to be (nor would I expect it to be) identical in content just the shapes should be identical. I'm imagining something like the pytorch `view` https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view function that allows me to make it whatever shape I would like but those type of function don't seem to be working with this array in Numpy. — David, Nov 30 '20 at 14:47

fountainhead · Accepted Answer · 2020-11-28T09:26:03.993

Here's a solution that does the job, but is guaranteed to be slow, because it is recursive, iterative, and vectorizes nothing at all:

import copy

# Generator function:
# Returns successive scalars from any crazily nested array
def arr_values(arr):
    for elem in arr:
        if (isinstance(elem, np.ndarray)):
            for sub_arr_elem in arr_values(elem):
                yield sub_arr_elem
        else:
            yield elem

# Generator function:
# Returns successive tuples (vector, index) from any arbitrarily nested array,
# such that assigning a value to vector[index] is the same as assigning the value
# to a position in the input array.
def arr_positions(arr):
    for pos,elem in enumerate(arr):
        if (isinstance(elem, np.ndarray)):
            for inner_arr_pos in arr_positions(elem):
                yield inner_arr_pos
        else:
            yield arr, pos

# Create a new array, having the shape of `b` (and incidentally, b's data also)
arr_in_shape_of_b = copy.deepcopy (b)

# Get the iterator for successive assignable positions in
# our destination array
iter_dest = arr_positions(arr_in_shape_of_b)

# Get the iterator for successive scalars from our source array
iter_src = arr_values(a)

# Now, iterate over the two iterators in tandem, and
# perform assignment of successive values from source,
# into successive positions in destination.
for ((dest_vec, pos), val) in zip(iter_dest, iter_src):
    dest_vec[pos] = val

Testing it out:

I tested with this demo data for a, and b:

a = np.array ([np.arange(6).reshape(2,3), np.array([np.arange(8).astype(np.ndarray),
                                                    23], dtype=np.ndarray),24], dtype=np.ndarray)
print (a.shape)
print (a)

b = np.roll(-1 * a, 2, axis=0)
print (b.shape)
print (b)

So, the input arrays a and b look like this:

(3,)
[array([[0, 1, 2],
       [3, 4, 5]])
 array([array([0, 1, 2, 3, 4, 5, 6, 7], dtype=object), 23], dtype=object)
 24]
(3,)
[array([array([0, -1, -2, -3, -4, -5, -6, -7], dtype=object), -23],
      dtype=object)
 -24 array([[ 0, -1, -2],
       [-3, -4, -5]])]

The output looks like this:

(3,)
[array([array([0, 1, 2, 3, 4, 5, 0, 1], dtype=object), 2], dtype=object) 3
 array([[ 4,  5,  6],
       [ 7, 23, 24]])]

Reshaping numpy array to be similar to different array with arbitrarily nested subarrays

1 Answers1