Prevent an iterator from being affected by changes to a variable used in its definition

Question

Edit/solution

Here's an example that more closely mimics my real problem, and a solution implementing the comment by jonrsharpe. Thanks!

import io
import heapq


# THIS DOES NOT WORK AS INTENDED
files = dict(f0=io.StringIO("first line\nsecond line"),
             f1=io.StringIO("FIRST LINE\nSECOND LINE"))
iterators = ((line.split() + [fname] for line in f)
             for fname, f in files.items())
list(heapq.merge(*iterators))
# [['FIRST', 'LINE', 'f0'],
#  ['SECOND', 'LINE', 'f0'],
#  ['first', 'line', 'f0'],
#  ['second', 'line', 'f0']]

# THIS DOES WORK
files = dict(f0=io.StringIO("first line\nsecond line"),
             f1=io.StringIO("FIRST LINE\nSECOND LINE"))
iterators = ((lambda x=fname: ((line.split() + [x]) for line in f))(fname)
             for fname, f in files.items())
list(heapq.merge(*iterators))
# [['FIRST', 'LINE', 'f1'],
#  ['SECOND', 'LINE', 'f1'],
#  ['first', 'line', 'f0'],
#  ['second', 'line', 'f0']]

Original question

How can I make the Python code below produce [(1, 0), (1, 1)] instead of [(2, 0), (2, 1)]? That is, I wish the result wasn't affected by changes to const made after the definition of the iterator it.

>>> const = 1
>>> var = range(2)
>>> it = ((const, i) for i in var)
>>> const = 2
>>> list(it)
[(2, 0), (2, 1)]

My real-world problem is to heapq.merge multiple files without keeping all their contents in memory. I'd like to programmatically generate iterators that report the file name together with each line of each file, e.g. ("file2.txt", "this is line 1"). Currently, all my iterators end up reporting the same filename.

You could create a new temporary variable for it to capture instead: `tmp = const` then `it = ((tmp, i) for i in var)`. — Blorgbeard, Nov 02 '17 at 18:00
Or use a `lambda` to create a new scope with that value set: `it = (lambda x=const: ((x, i) for i in var))(const)` — jonrsharpe, Nov 02 '17 at 18:02
So encapsulate the generator expression in a function? `const` is a closure, so yes, this is entirely expected. Generator expressions are no different from functions in this respect. Closures are 'live'. — Martijn Pieters, Nov 02 '17 at 18:08
You should write an answer with a solution (and then accept it), not edit it into the question. — River, Nov 02 '17 at 20:30
@River, thanks for letting me know. However, I don't see any "Your Answer" box at the bottom. Could there be anything preventing me from answering, e.g. the [duplicate] tag? — Jon Olav Vik, Nov 02 '17 at 20:50
@JonOlavVik oh yea, this is due to it being closed as a duplicate... I've rolled back my edit. Just keep in mind for future that answers belong in the answers section =) — River, Nov 02 '17 at 20:52

score 0 · Answer 1 · answered Nov 02 '17 at 18:08

You can move the generator expression from the module scope into a function, so the const value is read from the function scope and further changes to const do not influence the evaluation of the generator expression:

def hide_gen(const):
    return ((const, i) for i in [0, 1])

const = 1
it = hide_gen(const)
const = 2
print(list(it))
# [(1, 0), (1, 1)]

Prevent an iterator from being affected by changes to a variable used in its definition

Edit/solution

Original question

1 Answers1