python yield generator variable scope

Question

I am using yield to create a generator that returns chunks of a string that are being extracted using a regex and re.sub(). While I found an approach that worked, I am a bit confused about why it works one way but not another, as shown below:

This doesn't work (processchunk() is not assigning to the chunk declared in splitmsg):

def splitmsg(msg):
    chunk = None
    def processchunk(match):
        chunk = match.group(1)
        return ""
    while True:
        chunk = None
        msg = re.sub(reCHUNK,processchunk,msg,1)
        if chunk:
            yield chunk
        else:
            break

This does work (note the only difference being chunk is now a list chunks):

def splitmsg(msg):
    chunks = [ None, ]
    def processchunk(match):
        chunks[0] = match.group(1)
        return ""
    while True:
        chunks[0] = None
        msg = re.sub(reCHUNK,processchunk,msg,1)
        if chunks[0]:
            yield chunks[0]
        else:
            break

My question is basically why does it appear that the scoping of the chunk/chunks variable seem to depend on whether it is a plain variable or a list?

possible duplicate of [Python variable scope question](http://stackoverflow.com/questions/370357/python-variable-scope-question) — , Aug 24 '12 at 20:30

score 5 · Accepted Answer · answered Aug 24 '12 at 20:32

In python, variables can be 'pulled' from the surrounding scope if read from. So the following will work:

def foo():
    spam = 'eggs'
    def bar():
        print spam
foo()

because the variable 'spam' is being looked up in the surrounding scope, the foo function.

However, you cannot change the value of a surrounding scope. You can change global variables (if you declare them as global in your function), but you cannot do that for the variable spam in the above function.

(Python 3 changes this, it adds a new keyword nonlocal. If you define spam as nonlocal inside of bar you can assign to that variable a new value inside of bar.)

Now to your list. What happens there is that you are not altering the variable chunks at all. Throughout your code, chunks points to one list, and only to that list. As far as python is concerned, chunks the variable is not altered within the processchunk function.

What does happen is that you alter the contents of the list. You can freely assign a new value to chunks[0], because that's not the variable chunks, it is the list referred to by chunks, first index. Python allows this because it is not a variable assignment, but a list manipulation instead.

So, your 'workaround' is correct, if somewhat obscure. If you use Python 3, you can declare chunks as nonlocal within processchunk and then things will work without lists too.

@aaa90210: `chunk = match.group(1)` *shadows* the nonlocal variable `chunk`. — Steven Rumbalski, Aug 24 '12 at 20:35
@aaa90210: `chunk[0] = match.group(1)` is syntactic sugar for `chunk.__setitem__(0, match.group(1))`. It's not really an assignment even though it looks like it. — Steven Rumbalski, Aug 24 '12 at 20:38

score 1 · Answer 2 · edited May 23 '17 at 10:31

In the first case, you are creating a new local variable called chunk. A variable is treated as local to a function if you assign to it inside the function. In the second case, you are modifying the list referred to by the outer variable chunk. Because you don't assign to this variable, it's not treated as local. See for instance this previous question.

Assigning to a bare name in Python (someName = ...) is not the same as anything else; in particular it is not the same as item assignment (someName[0] = ...). The latter is calling methods under the hood to mutate the list.

python yield generator variable scope

2 Answers2