Need help understanding Python closures

Question

I have this code:

import re

def doReplace(toReplace):
    i = 1
    def chapterReplacer(_):
        result = 'Chapter %i' % i
        i += 1
        return result

    return re.sub('Chapter [a-zA-Z]+', chapterReplacer, test)

test = 'Chapter one Chapter Two Chapter three'
print doReplace(test)

when I run it, I get the following error:

Traceback (most recent call last):
  File "C:/Python26/replace.py", line 13, in <module>
    print doReplace(test)
  File "C:/Python26/replace.py", line 10, in doReplace
    return re.sub('Chapter [a-zA-Z]+', chapterReplacer, test)
  File "C:\Python26\lib\re.py", line 151, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "C:/Python26/replace.py", line 6, in chapterReplacer
    result = 'Chapter %i' % i
UnboundLocalError: local variable 'i' referenced before assignment

I was under the impression that chapterReplacer would capture the local variable i, but that doesn't seem to be happening?

related: http://stackoverflow.com/questions/141642/what-limitations-have-closures-in-python-compared-to-language-x-closures?rq=1 — jfs, Jun 15 '12 at 20:03
The dupe target is not that helpful here, because this is about nested functions trying to assign to a name from the parent scope. — Martijn Pieters, Nov 13 '13 at 13:37

Martijn Pieters · Answer 1 · 2012-06-15T20:19:53.467

Nope, and in python 2 you can't at all without resorting to using tricks with mutables:

def doReplace(toReplace):
    i = [1]
    def chapterReplacer(_):
        result = 'Chapter %i' % i[0]
        i[0] += 1
        return result

    return re.sub('Chapter [a-zA-Z]+', chapterReplacer, test)

Normally, python will only look in the surrounding scope for a variable if it is not being assigned to locally; as soon as the bytecompiler sees a direct assignment (i = something) and no global i statement to persuade it otherwise, a variable is considered local.

But in the above code we never assign to i in the chapterReplacer function. Yes, we do change i[0] but the value stored in i itself, a list, does not change.

In python 3, just use the nonlocal statement to have python look in it's closure for the variable:

def doReplace(toReplace):
    i = 1
    def chapterReplacer(_):
        nonlocal i
        result = 'Chapter %i' % i
        i += 1
        return result

    return re.sub('Chapter [a-zA-Z]+', chapterReplacer, test)

Because you are not re-assinging `i`; only members of `i`. The variable itself happens to be mutable, but because it's direct value (a list) doesn't change, python will look in the surrounding scope. — Martijn Pieters, Jun 15 '12 at 20:08

Hans Z · Accepted Answer · 2012-06-15T20:25:18.913

2

You can make i a function attribute

def doReplace(toReplace):
    chapterReplacer.i = 1
    def chapterReplacer(_):
        result = 'Chapter %i' % chapterReplacer.i
        chapterReplacer.i += 1
        return result

    return re.sub('Chapter [a-zA-Z]+', chapterReplacer, test)

EDIT: As of python 3, you can use nonlocal a la @MartijnPieters 's solution.

edited Jun 15 '12 at 20:25

answered Jun 15 '12 at 20:03

Hans Z

4,664
2
27
50

if chapterReplacer is only called within doReplace would it matter? – Hans Z Jun 15 '12 at 20:07
@Hans: yes, because every time you run `doReplace` you can get a new `chapterReplacer`. If you put the attribute on `doReplace`, then if you run `doReplace` again, you will be using the same `i`, which is incorrect – newacct Jun 15 '12 at 21:36
hmm @newacct but you're explicitly resetting the value every time when going into doReplace, so wouldn't it not matter? – Hans Z Jun 15 '12 at 21:38
@Hans: it would still matter if you have multiple threads or if the code inside could somehow recursively call `doReplace` or something like that – newacct Jun 15 '12 at 23:14
And it is subtleties like that that lead to hard-to-trace bugs, and why I do *not* use function attributes for closure-breaking variables! – Martijn Pieters Jun 17 '12 at 09:44

score 2 · Answer 3 · answered Jun 15 '12 at 20:05

2

In Python, if you assign to variable inside a function (even if with a compound assignment operator such as +=), that variable is considered local unless specified otherwise by a global or nonlocal statement.

answered Jun 15 '12 at 20:05

dan04

87,747
23
163
198

score 1 · Answer 4 · answered Jun 15 '12 at 20:06

1

When compiler sees that variable i gets another value inside function chapterReplacer it treats it as local, and no 'closure magic' is applied. If you remove line i += 1, your code will run.

answered Jun 15 '12 at 20:06

Roman Bodnarchuk

29,461
12
59
75

Need help understanding Python closures

4 Answers4

Linked