0

getattr(dir,"__name__") is dir.__name__ evaluates to False - is there an alternative to getattr that would yield True ?

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
user32531323
  • 13
  • 4
  • 1
    Please give a reproducible example. As far as I can tell the first expression should return `True`, and that is what I get in a few examples I have checked. Also, maybe please explain what you are trying to achieve with this, and why do you need `is` instead of just `==`. – jdehesa Jul 11 '19 at 14:56
  • 3
    It will return true if you use `==`. Why are you using `is` ? – khelwood Jul 11 '19 at 14:58
  • ```getattr(dir,"__name__") is dir.__name__``` Is literally the reproducible example, on what environment does it evaluate to ```True``` for you ? ```is``` is used here over ```==``` for precisely the reason ```is``` is used over ```==``` elsewhere - to test for identity rather than equality. – user32531323 Jul 11 '19 at 14:59
  • 2
    I don't think what you're asking for is possible. `dir.__name__` doesn't return the same object each time it is evaluated. Even `dir.__name__ is dir.__name__` evaluates to false. – khelwood Jul 11 '19 at 15:05
  • that is apparently correct and is quite astounding... – user32531323 Jul 11 '19 at 15:09
  • 1
    @user32531323: Not that astounding. The `__name__` "attribute" here is [actually a property](https://github.com/python/cpython/blob/3.7/Objects/methodobject.c#L193); rather than storing a full fledged Python `str`, it just makes one on demand when `__name__` is accessed. Reduces work/memory usage when `__name__` isn't accessed (which is common case). – ShadowRanger Jul 11 '19 at 15:52
  • @ShadowRanger, not a good reason - an implementation detail / optimization should not affect the semantics of operators ! – user32531323 Jul 11 '19 at 15:56
  • 2
    @user32531323: It's not affecting the semantics of operators. `is` tests by object identity. You have two different objects. You only use `is` when you have language guarantees of unique identities (e.g. for `None`, `NotImplemented`, `True`, `False`, and class types); no such guarantees are given for `__name__`. – ShadowRanger Jul 11 '19 at 16:00
  • Even javascript gets this right: ```f = function() { return 5; } f.name === f.name``` yields the expected ```true``` even though I'm (almost ?) sure V8 does not allocate an additional explicit String object to hold it. – user32531323 Jul 11 '19 at 16:00
  • @user32531323 what semantic property is it violating? this isn't astounding at all. – juanpa.arrivillaga Jul 11 '19 at 16:00
  • 1
    @user32531323: `===` isn't an identity test in JavaScript, it's a non-type coercing equality test. The equivalent Python test would be `dir.__name__ == dir.__name__`. [Read more here](https://stackoverflow.com/q/21774629/364696) and [here](https://stackoverflow.com/q/1504717/364696). – ShadowRanger Jul 11 '19 at 16:03
  • @ShadowRanger: fair enough re js, but having ```dir.__name__ is dir.__name__``` being false is still far from least-astonishment principle – user32531323 Jul 11 '19 at 16:14
  • @user32531323: Every `property` in Python potentially exhibits this feature. `is` is a power user tool in Python; Pythonic code written by all but the most advanced programmers should only be using it to compare to `None` and `NotImplemented` (though even `NotImplemented` is outside what most novice to intermediate programmers need). You need to default to testing with `==`, which would avoid this problem (and many related problems). If you *need* identity semantics, you can wrap your lookups with `sys.intern`; two interned strings with the same value are guaranteed to *be* the same string. – ShadowRanger Jul 11 '19 at 16:19
  • I suppose they could have implemented the getter to use `PyUnicode_InternFromString` instead of `PyUnicode_FromString` to get the behavior you expect, but that would make the initial lookup more expensive (and if the result was discarded, the intern entry would have to be removed and no work would be saved even in the future). – ShadowRanger Jul 11 '19 at 16:23
  • @ShadowRanger - this is also not the sort of thing that should make a difference in the behaviour of an identity operator at least to my unpythonic sensibilities. – user32531323 Jul 11 '19 at 16:44
  • @user32531323: I'm not sure you understand what identity means here. The identity operator behaves consistently; if the two objects are the *same* object (not the same value, actually the same object stored in the same memory), then `is` returns `True`. Your problem isn't with the identity operator, it's with properties, and specifically, that `__name__` was implemented as a property. All languages with properties have this behavior; they look like attributes, but since they're dynamically computed, they produce a value with a new identity each time the property is accessed. – ShadowRanger Jul 11 '19 at 16:55
  • If you don't like that `__name__` on functions happened to be implemented via a property, I get that. I suspect the choice was mostly about convenience/performance; the `PyMemberDef` structure is statically initialized in most use cases with C string literals, so they had to store the C string literal no matter what, and it would have slowed startup and increased the memory usage to *also* store the rarely used `PyString_Object`/`PyUnicode_Object`. The problem here is that your "traverse a graph of arbitrary Python objects" algorithm has trouble with distinguishing attributes from properties. – ShadowRanger Jul 11 '19 at 16:59
  • I understand that the is operator is consistent as defined, just that the definition is not very useful. The problem here is that the ```is``` operator is being applied on the **wrappers** of the returned values rather than the objects wrapped by them, the existence of these wrappers should not be visible to the programmer in such case afaic as it is entirely a python implementation effect. – user32531323 Jul 11 '19 at 17:19
  • @user32531323: To be clear, there are no wrappers involved here. The original C-style strings are stored in one place, which are read to initialize the Python level `str`, but that Python level `str` holds the complete data separate from the underlying C-style string. From the moment it is built, it no longer has anything to do with the memory holding the original C-style string. Python doesn't have a concept of anything like C++'s `std::stringview`; it can't actually make wrappers of the sort you describe for text data. – ShadowRanger Jul 15 '19 at 22:50

1 Answers1

4

The __name__ attribute of built-in functions is implemented (on the CPython reference interpreter) as a property (technically, a get-set descriptor), not stored as an attribute in the form of a Python object.

Properties act like attributes, but call a function when the value is requested, and in this case, the function converts the C-style string name of the function to a Python str on demand. So each time you look up dir.__name__, you get freshly constructed str representing the data; as noted in the comments, this means there is no way to have an is check pass; even dir.__name__ is dir.__name__ returns False, because each lookup of __name__ returned a new str.

The language gives no guarantees of how __name__ is implemented, so you shouldn't be assuming it returns the same object each time. There are very few language guaranteed singletons (None, True, False, Ellipsis and NotImplemented are the biggies, and all classes have unique identities); assuming is will work with anything not in that set when it's not an object you controlled the creation of is a bad idea. If you want to check if the values are the same, test with ==, not is.

Update to address traversing an arbitrary graph of python objects without getting hung up by descriptors and other stuff (like __getattr__) that dynamically generate objects (and therefore shouldn't be invoked to describe the static graph):

The inspect.getattr_static function should let you "traverse an arbitrary graph of python objects reachable from a starting one while assuming as little possible about the types of objects and the implementation of their attributes" (as your comment requested). When the attribute is actually an attribute, it returns the value, but it doesn't trigger dynamic lookup for descriptors (like @property), __getattr__ or __getattribute__. So inspect.getattr_static(dir, '__name__') will return the getset_descriptor that CPython uses to implement __name__ without actually retrieving the string. On a different object where __name__ is a real attribute (e.g. the inspect module itself), it will return the attribute (inspect.getattr_static(inspect, '__name__') returns 'inspect').

While it's not perfect (some properties may actually be backed by real Python objects, not dynamically generated ones, that you can't otherwise access), it's at least a workable solution; you won't end up creating new objects by accident, and you won't end up in infinite loops of property lookup (e.g. every callable can have __call__ looked up on it forever, wrapping itself over and over as it goes), so you can at least arrive at a solution that mostly reflects the object graph accurately, and doesn't end up recursing to death.

Notably, it will preserve identity semantics properly. If two objects have the same attribute (by identity), the result will match as expected. If two objects share a descriptor (e.g. __name__ for all built-in functions, e.g. bin, dir), then it returns the descriptor itself, which will match on identity. And it does it all without needing to know up front if what you have is an attribute or descriptor.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • How does one dynamically refer to the get-set descriptor itself rather than the result of its execution then ? – user32531323 Jul 11 '19 at 16:05
  • 1
    @user32531323: `type(dir).__dict__['__name__']` will get it, but that won't help you much; the descriptor is shared with all built-in functions/methods (it reads from a common field in the underlying C structure), so `type(bin).__dict__['__name__']` would return the same descriptor as `dir`, because `bin` is an instance of the same type (`PyCFunctionObject`). – ShadowRanger Jul 11 '19 at 16:08
  • 1
    @user32531323: It would help to know *why* you need to do this; the insistence on identity testing for something like `__name__` makes me worry you've got a [serious XY problem](https://meta.stackexchange.com/q/66377/322040). – ShadowRanger Jul 11 '19 at 16:10
  • It is in fact this shared reference I'm after, so thanks - marking as correct. – user32531323 Jul 11 '19 at 16:15
  • Basically I want to traverse an arbitrary graph of python objects reachable from a starting one while assuming as little possible about the types of objects and the implementation of their attributes – user32531323 Jul 11 '19 at 16:17
  • @user32531323: Is there a reason you can't just base the comparison on the `type` itself? `is` checks are valid for classes, so you could just compare `type(dir) is type(bin)` and the like. Trying to use `__name__` for the test is actually less specific than the implementing class after all. – ShadowRanger Jul 11 '19 at 16:21
  • Because I want to distinguish between instances not types, think of it as answering the question "how many distinct symbols are reachable ?" - ```__name__``` counts once for all the objects it appears in its traditional capacity but would be counted separately if it was somehow overridden in some class ```dir``` and ```bin``` are distinct symbols regardless. – user32531323 Jul 11 '19 at 16:29
  • However: ```type(dir).__dict__['__call__'] == type(dir.__call__).__dict__['__call__']``` is false (also when compared with ```is``` and ```id()```) Apparently because one is a ```slot wrapper '__call__' of 'method-wrapper' objects``` and the other is ```slot wrapper '__call__' of 'builtin_function_or_method' objects``` – user32531323 Jul 11 '19 at 16:42
  • @user32531323: Basically anytime you say `obj.methodname`, you're going to get a bound method-like object. Since `__call__` is a method, `dir.__call__` returns a bound method-like object wrapping `dir` (you can get back to `dir` itself by loading the `__self__` attribute of `dir.__call__`). `__call__` is effectively infinitely recursive; you can keep accessing it over and over, and it'll just get wrapped over and over. `type(dir)` is telling you about `dir` itself; `type(dir.__call__)` is telling you about the bound-method like wrapper dynamically constructed around `dir`. – ShadowRanger Jul 11 '19 at 16:49
  • @user32531323: indeed and it is breaking this recursion without doing the un-binding or prototype analysis myself I'm concerned with - fwiw it seems to be finite when accessing through the type __dict__: once a 'builtin_function_or_method' is reached it's type().__dict__['__call__'] is itself but that might be accidental/implementation-dependent ? – user32531323 Jul 11 '19 at 16:55
  • @user32531323: Check my updated answer. I suspect what you need is `inspect.getattr_static`, which won't cause infinite recursion, and will return descriptors themselves without invoking them, while returning actual attributes normally. – ShadowRanger Jul 11 '19 at 17:17