1

I would like to make all possible combinations with elements from 5 different arrays. However, when I ran my codes, the error showed the number was too high. Would there be a solution to this problem?

Here is my code:

import numpy as np

a=np.arange(1000000000)
b=np.arange(2000000000)
c=np.arange(2000000000)
d=np.arange(1000000000)
e=np.arange(1000000000)

from itertools import product

iterables = [a,b,c,d,e]
total_combi_index=[]
for t in product(*iterables):
    total_combi_index.append(t)
Cielo Kim
  • 21
  • 4
  • 3
    Why do you want to do that? That would use a lot of memory, probably more than you have. – syntonym Dec 01 '20 at 17:10
  • @syntonym what would be the upper limit normally? With my codes above, it didn't work, so I ended up choosing random number out of each arrays, but when I chose 100, the error occurred again. – Cielo Kim Dec 01 '20 at 17:21
  • For the maximal size of a list in python see [this answer](https://stackoverflow.com/questions/855191/how-big-can-a-python-list-get). But is this really what you want? Why do you want to create such a big list? What are you doing with the numbers afterwards? – syntonym Dec 01 '20 at 17:39
  • @syntonym Thanks! I'm trying to find 10-20 variations of 4D vectors in a close distance to a certain 4D vector. Even with 6,250,000 combinations, the nearest-distance vector was far off, so I am trying to increase total number of combinations. – Cielo Kim Dec 01 '20 at 17:49
  • 1
    You don't need to create a list to do that, your `product(*iterables)` already is an iterable that holds all combinations. Can you not just iterate over that iterable? (unrelated, but that doesn't seem the best way to find a 4D vector that is close to a different 4D vector). – syntonym Dec 01 '20 at 17:53
  • @syntonym Do you mean there is no need to create the list 'total_combi_index' and append the element into the list? In that case, I can't access the element. I tried to save 'list(product(*iterables)' with a variable, but the error said ' 'itertools.product' object is not callable.' – Cielo Kim Dec 01 '20 at 19:55
  • Yes, that's exactly what I meant. You also don't need the `list` around the product, you only need `product(*iterables)` when you want to iterate through it. You probably called the result product: `product = product(*iterables)`, but it's hard to say without the actual code. Can you update the question with the code you're actually trying to run? Otherwise it's very hard to answer your question in a helpful way. – syntonym Dec 01 '20 at 21:13

1 Answers1

0

we have standard lib:itertools chain method and from_iterable method:

itertools.chain(*iterables) Make an iterator that returns elements from the first iterable until it is exhausted, then proceeds to the next iterable, until all of the iterables are exhausted. Used for treating consecutive sequences as a single sequence. Roughly equivalent to:

def chain(*iterables):
    # chain('ABC', 'DEF') --> A B C D E F
    for it in iterables:
        for element in it:
            yield element

classmethod chain.from_iterable(iterable) Alternate constructor for chain(). Gets chained inputs from a single iterable argument that is evaluated lazily. Roughly equivalent to:

def from_iterable(iterables):
    # chain.from_iterable(['ABC', 'DEF']) --> A B C D E F
    for it in iterables:
        for element in it:
            yield element

in python, we use iterator to resolve this kinds of problem

Han.Oliver
  • 525
  • 5
  • 8