"except Foo as bar" causes "bar" to be removed from scope

Question

Given the following code:

msg = "test"
try:
    "a"[1]
except IndexError as msg:
    print("Error happened")
print(msg)

Can somebody explain why this causes the following output in Python 3?

Error happened
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    print(msg)
NameError: name 'msg' is not defined

score 54 · Accepted Answer · edited Oct 24 '18 at 18:23

msg in the except clause is in the same scope as msg on the first line.

But in Python 3 we have this new behavior too:

When an exception has been assigned using as target, it is cleared at the end of the except clause. This is as if
except E as N:
    foo
was translated to
except E as N:
    try:
        foo
    finally:
        del N
This means the exception must be assigned to a different name to be able to refer to it after the except clause. Exceptions are cleared because with the traceback attached to them, they form a reference cycle with the stack frame, keeping all locals in that frame alive until the next garbage collection occurs.

so, you "overwrite msg" in the exception handler, and exiting the handler will delete the variable to clear the traceback reference cycle.

This answers specifically the question how and why the previous `msg` gets deleted after the `except:`, including reference to the docs, so IMHO it should be the accepted answer. — Jeronimo, Oct 24 '18 at 10:56

score 34 · Answer 2 · answered Oct 24 '18 at 10:01

34

Yes, as soon as the exception is raised and msg is assigned with the new exception object, the original object has no more reference and is therefore deleted. The new exception object is also deleted as soon as it leaves the except block.

You can verify it by overriding the __del__ method of the object and the exception assigned to msg:

class A:
    def __del__(self):
        print('object deleted')
class E(Exception):
    def __del__(self):
        print('exception deleted')
msg = A()
try:
    raise E()
except E as msg:
    print("Error happened")

This outputs:

object deleted
Error happened
exception deleted
NameError: name 'msg' is not defined

answered Oct 24 '18 at 10:01

blhsing

91,368
6
71
106

It's much simpler to use `weakref.ref` to demonstrate this; `__del__` has confusing semantics which vary between runtimes and python versions. – habnabit Oct 24 '18 at 10:39
5

@LightnessRacesinOrbit Yeah, this is definitely strange. I mean: if you want a new scope then create a new scope and don't kill the reference to the first `msg`. If you don't want a new scope, then why kill the exception reference? – Giacomo Alzetta Oct 24 '18 at 12:00
6

It's not wonky at all. By using `except ... as msg` you're binding `msg` to the exception, which removes all references to the original object. Conversely, if `except ... as m` was used, it wouldn't result in any issue since the original binding `msg` isn't modified in the except block. – Shamtam Oct 24 '18 at 12:45
31

@Shamtam As Giacomo explains, it’s *definitely* wonky. The way Python works, it rebinds `msg` *in the existing scope*. But the scope of `msg` exists beyond the `try` block (and would continue to, if no exception were raised). The fact that a *runtime condition* determines the scope of a variable (which should be static) is a crass subversion of the type system. Since Python is dynamically typed this of course works but it’s nevertheless weird. What’s more, Python 2 handles this completely differently, and as expected. – Konrad Rudolph Oct 24 '18 at 13:18
@Shamtam It's wonky in the sense that it is unintuitive and does not match other popular languages (though whether this is itself "bad" is another debate) or even the rest of the same language – Lightness Races in Orbit Oct 24 '18 at 13:22
@Shamtam except that's not how scoping works. There *is* a reference to the original object, it's a couple lines later (remember, scope is extent in code-space, not extent in time). The problem seems to be that Python is *failing* to create a scope. – hobbs Oct 24 '18 at 13:23
3

@hobbs try-catch does not introduce a new scope, so no, after the exception block is run, there is no longer a reference to the original object that msg was bound to, since msg was rebound and then deleted by the except block. As @Konrad Rudolph explains, it's the way Python works. It's the same idea as `x = A()\ x = B()\ del x\ print x`. The behavior in Python2 is different than this, but still "unexpected." Instead of throwing a `NameError`, you simply get `string index out of range`. The difference between Py2 and Py3 is that in Py2 the exception is not deleted (see Uku Loskit's answer). – Shamtam Oct 24 '18 at 13:37
@LightnessRacesinOrbit Agreed about the "bad" debate. I suppose it's not counter-intuitive to me since I primarily work in Python, and make it a habit to not create any bindings in an except block that have the same name as any other bindings. I don't find it counter-intuitive that that try-catch doesn't introduce new scope, though. I think that's simply what one is used to. – Shamtam Oct 24 '18 at 13:40
2

This is a good reason to be semantic about using the term "binding" instead of "variable" when describing behavior in Python that would be unexpected in other languages. – Shamtam Oct 24 '18 at 13:44
11

@Shamtam "it's the way Python works": that is surely true. Nobody is denying that. But it doesn't mean that people shouldn't consider Python to be wonky. Scope rules in "other popular languages" allow programmers using those languages to avoid the mental burden of keeping track of the names they've used before they got to the except block (or analogous) so as to be able to avoid using existing names in the block; the interpreter or compiler does it for them. – phoog Oct 24 '18 at 14:38
4

@Shamtam What is the difference between `msg=1;try:... except E as msg:` and `x=1;for x in something:...`? At the end of the `for` `x` keeps the binding to the last iteration, I don't see any good reason to add a hidden `del msg` at the end of the `except` block. Python **never** worked in this way, **all other statements do not work in this way**, previous versions of python did not work in this case. As I said, from a "lexical-scope point of view" either you introduce a new scope (and `msg` keeps the old value) or you just rebind the name. Deleting a binding in that way shouldn't be possible – Bakuriu Oct 24 '18 at 16:28
@Bakuriu The difference is that in the context of an exception, after it has been handled, it was decided in Py3 to delete the exception. Py2 still has "unexpected" behavior with this same code, where after the except block is through, `print msg` will return `string index out of range` instead of `test` like you'd expect if you had traditional language scoping, so I don't agree with saying that "Python **never** worked this way." Again, the original object `"test"` which was bound to `msg` is deleted as soon as the exception is thrown at runtime, regardless of Python version. – Shamtam Oct 24 '18 at 16:49
@Bakuriu In the case of a for loop, you _still_ aren't in a different scope. In Python, if you create a binding inside of a for loop, then try printing it outside of the for loop, you won't have any error. Try that in "other popular languages" and you'll get an undefined variable error. It's a nuance of Python's fundamental binding/object structure. Perhaps that unsafe runtime dynamic behavior is wonky when compared to other languages, but it can be useful sometimes. Python doesn't scope like other languages, simple as that. (This should be in chat, but the auto-move link hasn't appeared...) – Shamtam Oct 24 '18 at 16:56
5

@Shamtam The python2 behaviour is what **I expect**. If `except` does not create a new scope the old `msg` *should be gone after reassigning it to the exception*, **but there is no reason to delete the exception**. We could easily do the same with `for` loop: add a hidden `del ` at the end of the `for` (or the `else` block attached to it). Same with `with`: `with something as x: ...; print(x)`. They decided to add this hidden `del` for no reason breaking all the conventions python already had. You are justifying this simply saying "because it's written in the standard", but ... – Bakuriu Oct 24 '18 at 17:20
2

@Shamtam you did not give any rationale as why that decision was taken, given that it surely breaks the least astonishment rule. Maybe there was a reason why is not a good idea to have access to the exception object outside the `except` block, but nothing you have written gives any hint as to why that should be the case. – Bakuriu Oct 24 '18 at 17:21
10

The reason `msg` is deleted at the end of the `except` block has nothing to do with scope. They wanted to clear a reference cycle created by the addition of the `__traceback__` attribute in Python 3. See [PEP 3110](https://www.python.org/dev/peps/pep-3110/#semantic-changes). It's definitely confusing, and there's a strong argument to be made that Python should stop trying to support running with the GC off, but it's not a scope issue. – user2357112 Oct 24 '18 at 18:27
@Bakuriu And it seems others would expect that `msg` should still be `"text"` like it would in other languages (due to scope). I personally think that is much more intuitive than it holding the exception as in Py2, coming from other languages where there actually is a scope change when handling exceptions. My whole point here is that there is no scope like in a traditional language; I'm not defending anything about Python 3's rationale about deleting exceptions after processing. All I'm saying is that the "scope-less" behavior is a nuance of Python bindings, nothing more. – Shamtam Oct 24 '18 at 20:09
@user2357112 That's good information. AFAICT they could have also decided to use a more complex translation that checked if the `as` target already existed and, if so, stored the reference in a new binding and after the "`del msg`" they could restore the old value. This would be the right way of doing it if keeping the exception around is an issue. They decided to implement just a rough adaptation to fix the issue which produces this wonky behaviour. Sure my proposal would too be a deviation from the "pureness" of scoping rules but I fell it's much less astonishing than the current situation – Bakuriu Oct 24 '18 at 22:15
@user2357112 Am I reading that right? Python changed the semantics of the language in a nonsensical way to work around a buggy GC? If so, good grief – Alex Celeste Oct 25 '18 at 10:28
2

But why the message is "Name *msg* not defined", while it's just a referenced object what became inaccessible? Name *msg* is still defined tho, isn't it? – Maciek Oct 25 '18 at 10:50
@Maciek: Python only distinguishes between unbound and nonexistent variables for function locals. At global scope (which the example's code runs in), a deleted name and a name that was never defined are treated the same. – user2357112 Oct 25 '18 at 15:10

score 7 · Answer 3 · answered Oct 24 '18 at 10:13

Exception blocks delete the caught variable at the end of the block, but they do not have their own scopes. So the sequence of events goes:

1) msg is set to some string in the local scope

2) msg is set to an IndexError object in the same local scope as 1

3) msg is deleted from the local scope when the Exception block ends

4) msg is no longer defined in the local scope, so the attempt to access it fails

"except Foo as bar" causes "bar" to be removed from scope

3 Answers3

Linked