Python: Creating an N-dimensional list from separate lists of varying sizes and dtypes

Question

Say I have these lists:

a = [1, 2, 3, 4]
b = [6,7]
c = ['a', 'b', 'c']

I would like to create a 3-Dimensional data structure that contains new lists combining all the elements of each list together in all possible ways. I would then like to access these new list using matrix like indexing.

So for example, say f is a function that does what I want. Then I could do this:

m = f(a,b,c)

Then m[0][0][0] would give [1,6,'a'], m[1][0][0] would give [2,6,'a'], and so on.

Basically, I know I can already do this using nested for loops.

jList = []
for j in a:
    kList = []
    for k in b:
        lList = []
        for l in c:
            o = [j,k,l]
            lList.append(o)
        kList.append(lList)
    jList.append(kList)

This gives me a list of lists of lists of lists.

[[[[1, 6, 'a'], [1, 6, 'b'], [1, 6, 'c']], 
  [[1, 7, 'a'], [1, 7, 'b'], [1, 7, 'c']]], 
 [[[2, 6, 'a'], [2, 6, 'b'], [2, 6, 'c']], 
  [[2, 7, 'a'], [2, 7, 'b'], [2, 7, 'c']]], 
 [[[3, 6, 'a'], [3, 6, 'b'], [3, 6, 'c']], 
  [[3, 7, 'a'], [3, 7, 'b'], [3, 7, 'c']]], 
 [[[4, 6, 'a'], [4, 6, 'b'], [4, 6, 'c']], 
  [[4, 7, 'a'], [4, 7, 'b'], [4, 7, 'c']]]]

This is fine, but I have to do this for dimensions higher than 3, and the many nested for loops just seems like it can't possibly be the best or most efficient way to build this type of data structure. I can't help but thinking there must be a better solution, something in some library like the function I made up above, maybe numpy function, but I've done a lot of searching and have not found something for lists of varying sizes and data types like I have.

Any ideas?

`list(itertools.product(a, b c))` won't give you the nesting you're looking for, but otherwise it seems perfect. Also, this is easily expandable. `lists = [a, b, c]` then call `list(itertools.product(*lists))`. If you later want to add a fourth iterable you can do `lists.append(d)` and call `list(itertools.product(*lists))`. — Steven Rumbalski, Jul 25 '17 at 18:59
@cᴏʟᴅsᴘᴇᴇᴅ Isn't the grouping into sublists supposed to be part of the problem? — Moses Koledoye, Jul 25 '17 at 19:01
@cᴏʟᴅsᴘᴇᴇᴅ: Not an exact duplicate as the OP wants a differently shaped result. — Steven Rumbalski, Jul 25 '17 at 19:01

Moses Koledoye · Answer 1 · 2017-07-25T19:14:06.073

You can use itertools.product to take the cartesian product of the lists, then use numpy.reshape for reshaping, taking the last dimension from the number of lists, then to the length of the innermost list and up until the length of the outermost:

>>> args = a,b,c
>>> np.array(list(product(*args))).reshape(len(a), len(b), len(c), len(args))
array([[[['1', '6', 'a'],
         ['1', '6', 'b'],
         ['1', '6', 'c']],

        [['1', '7', 'a'],
         ['1', '7', 'b'],
         ['1', '7', 'c']]],


       [[['2', '6', 'a'],
         ['2', '6', 'b'],
         ['2', '6', 'c']],

        [['2', '7', 'a'],
         ['2', '7', 'b'],
         ['2', '7', 'c']]],


       [[['3', '6', 'a'],
         ['3', '6', 'b'],
         ['3', '6', 'c']],

        [['3', '7', 'a'],
         ['3', '7', 'b'],
         ['3', '7', 'c']]],


       [[['4', '6', 'a'],
         ['4', '6', 'b'],
         ['4', '6', 'c']],

        [['4', '7', 'a'],
         ['4', '7', 'b'],
         ['4', '7', 'c']]]],
      dtype='<U21')

You can generate the shape on the fly by using: tuple(map(len, args)) + (len(args),) (suggested by @StevenRumbalski), so that you now have:

>>> np.array(list(product(*args))).reshape(tuple(map(len, args)) + (len(args),))

Or:

>>> np.array(list(product(*args))).reshape(tuple(map(len, args)) + (-1,))

Without needing to specify the last dimension.

`new_shape = list(map(len, args)) + [len(args)]` then `.reshape(*new_shape)` would make this generic. — Steven Rumbalski, Jul 25 '17 at 19:07

score 0 · Answer 2 · answered Jul 25 '17 at 19:21

Vanilla Python.

from itertools import product

a = [1, 2, 3, 4]
b = [6,7]
c = ['a', 'b', 'c']
data = [a,b,c]
dims = [range(len(i)) for i in data]
keys = product(*dims)
vals = (list(val) for val in product(*data))
space = dict(zip(keys,vals))
for key in space:
    print "{} -> {}".format(key, space[key])

(3, 0, 2) -> [4, 6, 'c']
(0, 1, 1) -> [1, 7, 'b']
(0, 1, 2) -> [1, 7, 'c']
(1, 0, 1) -> [2, 6, 'b']
(1, 0, 0) -> [2, 6, 'a']
(2, 0, 1) -> [3, 6, 'b']
(2, 0, 0) -> [3, 6, 'a']
(3, 1, 0) -> [4, 7, 'a']
(3, 1, 1) -> [4, 7, 'b']
(0, 0, 2) -> [1, 6, 'c']
(2, 0, 2) -> [3, 6, 'c']
(0, 0, 1) -> [1, 6, 'b']
(0, 0, 0) -> [1, 6, 'a']
(2, 1, 2) -> [3, 7, 'c']
(1, 1, 1) -> [2, 7, 'b']
(1, 0, 2) -> [2, 6, 'c']
(1, 1, 0) -> [2, 7, 'a']
(2, 1, 0) -> [3, 7, 'a']
(2, 1, 1) -> [3, 7, 'b']
(1, 1, 2) -> [2, 7, 'c']
(3, 0, 0) -> [4, 6, 'a']
(3, 1, 2) -> [4, 7, 'c']
(3, 0, 1) -> [4, 6, 'b']
(0, 1, 0) -> [1, 7, 'a']

Python: Creating an N-dimensional list from separate lists of varying sizes and dtypes

2 Answers2