how do you chain iterables?

Question

I have two iterable objects and want to chain them, i.e. concatenate them. Specifically, being iterable the objects return an iterator, and this can be repeated. The concatenation of these two iterables should be an iterable that gets the two iterators from the inputs each time and returns the concetenated iterator. I am still fairly new to Python and find it very subtle and quite difficult - not like the "easy" language everyone tells me it is. But I would have thought there'd be a simple way to do this particular task. Note chain from itertools doesn't work:

from itertools import chain

def i():
    n = 0
    while n < 2:
        yield n
        n = n+1

def j():
    m = 0
    while m < 3:
        yield m
        m = m+1

print("iterating i")        
for x in i():
    print(x)
    
print("iterating i again")   
for x in i():
    print(x)
    
k = chain(i(),j())

print("iterating k")    
for x in k:
    print(x)
    
print("iterating k again")   
for y in k:
    print(y)
print("but it's empty :(")

giving

iterating i
0
1
iterating i again
0
1
iterating k
0
1
0
1
2
iterating k again
but it's empty :(

Here chain seems to operate on the iterators giving an iterator, but I want to chain two iterables giving an iterable.

Responses to initial comments:

I don't think this is a question about generators: I just used generators to illustrate.

Some people have said the iterable has run out of stuff. But as I understand it, that is not right: the iterable can always make new iterators. You can see that i() can be repeated. One particular iterator has been exhausted. This question is about the difference between iterator and iterable.

Finally, of course I could save the data in a list and iterate as many times as I like over that list. But I want you to imagine that 2 and 3 are so huge that this is impractical.

Thanks in advance :)

Answer 1

That is how generators are expected to work. In Layman's terms, after you used it once, its empty. The idea is not to use new memory to store stuff but to create stuff on the run

When you do k = chain(i(),j()) you create the generator object k and in the first for x in k loop, you empty it. Now when you try to loop through the same k in for y in k loop, there's nothing left to iterate in the iterator k

In your first for x in i() loop, you call i() and create a new generator there for looping. Doing that again in for y in i(), you're looping over a new generator; not the old one you used previously

If you want to use k multiple times, you can create a list instead of a generator

k = list(chain(i(), j()))

---

EDIT: What you want to do is simply use for x in chain(i(), j()) and for y in chain(i(), j()) instead of for .. in k if you don't want a list. This will create a new generator object at each for loop. k = chain(i(), j()) creates only one generator and you're trying to use it two times

Answer 2

Like I said, Python is subtle. And it seems it really is tricky to explain what I want, let alone do it!

I have some code now that does do what I want. It's not particularly pretty. I have removed the stuff on generators because this seemed to be a red-herring. If this is really the simplest way of doing it in Python then it is the answer to my question. But someone out there might know a simpler way.

from itertools import chain

# make iterators without using generators
class iterthing:
    def __init__( self , n ):
        self._max = n
    def __iter__( self ):
        return iter(range(self._max))
    
i = iterthing(2)
j = iterthing(3)

print("iterating i")        
for x in i:
    print(x)

print("iterating j")        
for x in j:
    print(x)
    
print("iterating i again")   
for x in i:
    print(x)

# join two iterables
class metachain:
    def __init__( self, i , j ):
        self._first = i
        self._second = j
    def __iter__( self ):
        return chain( iter(self._first), iter(self._second) )
    
k = metachain(i,j)

print("iterating k")    
for x in k:
    print(x)
    
print("iterating k again")   
for y in k:
    print(y)
print("this time it repeats as required with no lists stored in memory")

Answer 3

You can do some magic via defining your own chain which expects functions:

def mychain(f1,f2):
  return itertools.chain(f1(),f2())

and then using functools.partial() to create your k:

k=functools.partial(mychain,i,j)

for x in k():
  print(x);
print("again")
for x in k():
  print(x)

(of course then you can make it *args, just the idea may be more readable with 2 arguments)