0

I have the following code:

from itertools import product
from joblib import Parallel, delayed
from operator import mul
import numpy as np

lst = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]
arr = np.array(lst)

flat = np.ravel(arr).tolist()
lst = (list(a) for a in product(flat, repeat=2))
results = Parallel(n_jobs=-1)(delayed(mul)(x, y) for x, y in lst)

rows = arr.shape[0]
cols = arr.shape[1]

arr3d = np.array(results).reshape(rows, cols, rows, cols)

def stacking():
    final_arr = np.empty((1, cols*cols))   # Initialize final_arr
    for i in range(0,rows):
        first_block = arr3d[i,0]   # Initialize first_block
        for j in range(1,cols):
            first_block = np.column_stack((first_block, arr3d[i,j]))
        final_arr = np.row_stack((final_arr, first_block))
    yield np.delete(final_arr, 0, axis=0)

next(stacking())

I wish to convert both the for loops (i's and j's) to a generator. When I tried to do this for the j loop:

def stacking():
    final_arr = np.empty((1, cols*cols))   # Initialize final_arr
    for i in range(0,rows):
        first_block = arr3d[i,0]   # Initialize first_block
        first_block = (np.column_stack((first_block, arr3d[i,j])) for j in range(1,cols))
        final_arr = np.row_stack((final_arr, first_block))
    yield np.delete(final_arr, 0, axis=0)

next(stacking())

I get the following error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-35-ba342fd13303> in <module>
----> 1 next(stacking())

<ipython-input-34-303528d26770> in stacking()
      4         first_block = arr3d[i,0]   # Initialize first_block
      5         first_block = (np.column_stack((first_block, arr3d[i,j])) for j in range(1,cols))
----> 6         final_arr = np.row_stack((final_arr, first_block))
      7     yield np.delete(final_arr, 0, axis=0)

<__array_function__ internals> in vstack(*args, **kwargs)

~\Anaconda3\lib\site-packages\numpy\core\shape_base.py in vstack(tup)
    281     if not isinstance(arrs, list):
    282         arrs = [arrs]
--> 283     return _nx.concatenate(arrs, 0)
    284 
    285 

<__array_function__ internals> in concatenate(*args, **kwargs)

ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 9 and the array at index 1 has size 1

Is there a way to use np.column_stack/np.row_stack together with generators?

I am basically trying to make this piece of code run faster and use less memory, as I am dealing with large arrays. If there is another way to tweak this piece of code to do this, please let me know! Many thanks.

Leockl
  • 1,906
  • 5
  • 18
  • 51
  • The only thing you should do in a loop is list append. All `stacking` is done once, outside the loop. – hpaulj Mar 07 '20 at 05:40
  • Generators only help if you feeding one list on to another and so forth. They aren't useful when working with arrays. – hpaulj Mar 07 '20 at 05:43
  • Ok. Is there any other way I could optimise this code for speed and memory? Or is it optimal already? – Leockl Mar 07 '20 at 06:08
  • 1
    So you start with a `final_arr = np.empty((1, cols*cols))` array - one row of junk. Then add a bunch to it, and at the end `np.delete` that junk! Why didn't you start with `np.empty((0, cols*cols))`? Better yet start with `alist=[]`, append a bunch to it, and then `column_stack` the list. `concatenate` and all the `stack` variations (except `np.append`) accept a list. Take advantage of that! – hpaulj Mar 07 '20 at 06:08
  • Thanks for the tip on `np.empty((0, cols*cols))`! If I use `list=[]`, I wouldn't have the option to use `reshape()` on a list. This code is basically doing the Kronecker (tensor) product. I am trying to write my own code to do the Kronecker product because my understanding is that `np.kron()` is not written for parallel processing and minimizing memory – Leockl Mar 07 '20 at 06:35
  • Let's step back a minute. Do you understand the error message? To use `concatenate` (or any of the `stack` functions that use it), you have to understand array dimensions; you can't be sloppy. Also remember, when doing `x = np.concatenate((x, y))` that you are making a new array, copying values from both `x` and `y`. If you do this in a loop, the size of that copy gets bigger and bigger. There's no memory or time savings! – hpaulj Mar 07 '20 at 08:17
  • 1
    https://stackoverflow.com/questions/60383660/why-is-numpys-kron-so-fast/60386471#60386471 – hpaulj Mar 07 '20 at 08:18
  • Thanks @hpaulj, the link was really helpful. The time savings in my code would be the Parallel step and the memory savings would be in the def-yield generator – Leockl Mar 07 '20 at 08:48

0 Answers0