1

Preface: the question is to understand python internals, so pls don't answer with 'upgrade python' or 'import six'

Let take for example, 'print'

To replace the 'print' builtin, I can write my python function.

python 2.6 : at runtime, I get syntax error

solution

from __future__ import print_function

Now I can redefine print. Why ? I think because now my print declaration is the same as the new print function.

python 2.4 : at runtime, I get syntax error

There isn't

from __future__ import print_function

So, for print (or any other builtin): is it possible to monkey patch with a new function, with a different declaration - args and kwargs ?

In other words.

from __future__ import print_function substitutes the keyword with a function. I want to understand HOW, in general. Please don't be focused exclusively on print, it is an handy example.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
Massimo
  • 3,171
  • 3
  • 28
  • 41
  • 1
    In Python 2, `print` was a keyword you could not assign to. In Python 3, `print` is "just" a function which a name you can reassign. `__future__` imports are a little bit of magic. – timgeb Sep 17 '18 at 19:54
  • not a pro here, but `from future import print` tellls the interpreter to load a python 3 stlye `print function` - for python 2x print was not a function but a protected keyword that is protected from being redefined – Patrick Artner Sep 17 '18 at 19:54
  • Side-note: The docs claim `from __future__ import print_function` [was new in 2.6.0a2](https://docs.python.org/2/library/__future__.html), so it's a little strange that it works in 2.5... – ShadowRanger Sep 17 '18 at 20:05
  • @ShadowRanger Thnks, you're right, I confuse with 'with_statement'. My question still stands. – Massimo Sep 17 '18 at 20:09
  • 4
    You say "Please don't be focused exclusively on `print`, it is an handy example." but there is literally nothing else remotely like the case of `print` on Python 2. In no other scenario does a `__future__` import change a keyword to a non-reserved word (some imports change non-reserved words to keywords, but `print_function` is the only case going from keyword to non-keyword). – ShadowRanger Sep 17 '18 at 20:13

2 Answers2

3

In Python 2, print is a keyword. You can't redefine print for the same reason you can't redefine if, while, def, or any other language keyword; they're part of the language syntax, and they're handled by the parser, not by looking up a function.

from __future__ import print_function is also handled by the parser. It looks like a normal import, and it does actually import a thing, but the thing it imports has nothing to do with the statement's primary functionality, which is to tell the parser to stop treating print as a keyword.

With the keyword disabled, print is treated as an ordinary name following ordinary name lookup rules. Such a lookup finds the print built-in function, which is normally hidden by the keyword.

You cannot define your own magic imports; they have to be built into the interpreter itself. Since you cannot define your own magic imports, and since no other __future__ imports turn keywords into non-keywords, there's no generalization to be done.

(People will sometimes say that builtins like list or dict are keywords. They're not keywords; those people are using the word "keyword" incorrectly.)

As for monkey-patching built-in functions (not keywords) with a different signature from the original, you can do that. It's probably a bad idea, but you can do it. The process is identical to how you'd monkey-patch a built-in normally.

user2357112
  • 260,549
  • 28
  • 431
  • 505
  • 2
    Amusing implementation detail: Once `from __future__ import print_function` is available, even when it's not activated, `print` *is* a builtin (you can access it with `vars(__builtins__)['print']`). It's just that unless the future feature is active, `print` being a keyword takes effect before LEGB lookup, so you never try to look up the name `print` unless you're using weird hacks to pull it from the builtins by string name. – ShadowRanger Sep 17 '18 at 20:03
  • 'from __future__ import print_function' substitutes the keyword with a function. I want to understand HOW, in general. pls don't be focused exclusively on print, it is an handy example. – Massimo Sep 17 '18 at 20:04
  • @Massimo: The case of `from __future__ import print_function` is unique; no other built-in requires that sort of rigmarole, because `print` is unique in that, on Py2, it defaults to being a keyword statement, like `del`, `return`, etc., not a function at all. That's why you can't normally replace it. Everything else is either always a keyword statement (can never be replaced) or always a built-in function (can always be replaced, though it's often a bad idea). – ShadowRanger Sep 17 '18 at 20:09
  • 2
    But it's not general. It's a specific hack in the parser that turns off the keyword. There is no general version of it. – Max Sep 17 '18 at 20:11
  • `from __future__ import ...`, despite sharing the same syntax, is *not* a regular import. It fundamentally alters the language recognized by the parser (which is why it *must* be the first statement in a module). Note that `getattr(__builtins__, 'print')("hi")` works without the `__future__` import. – chepner Sep 17 '18 at 20:23
3

All from __future__ import print_function does is toggle a flag in the parser to make it stop treating print as a keyword; once that's done, references to print seamlessly become like references to any other non-keyword that constitutes a legal variable name (they go through LEGB lookup, and get found in the B of LEGB, the builtins scope). This behavior is hard-coded deep in the Python interpreter; there is no way to achieve a similar effect for any other keyword without building a custom version of Python, or engaging in some other egregious hackery well beyond the scope of any reasonable problem.

As of 2.6, __builtin__ has a print function on it, so any module that uses from __future__ import print_function (and can therefore reference the name print as opposed to the keyword print) will see __builtin__.print (if it hasn't been shadowed by something in the local, nested or global scope). It's still there for every module, but in modules without the __future__ import, references to print are resolved to the keyword at compile time and replaced with the raw bytecode implementing the special print statement (the same way del and return behave; you can't name a variable either of those things for the same reason, they're keywords), so modules without the import never get a chance to lookup the print function.

This doesn't generalize to other cases, because __future__ has no other cases where one of its features converts a keyword to a non-keyword. For all other actual built-in functions, being able to overwrite them on a per-module basis is as simple as assigning to the name at global scope (shadowing it for that module), e.g.:

def abs(x):
    return x  # Who needs absolute value anyway?
# From here on out, references to abs in this module see your override, not the built-in

While it's possible to reassign the built-in globally, it's a terrible idea (since every other module that uses it likely relies on the built-ins original behavior). That said, it's not hard:

import __builtin__

def abs(x):
    return x  # Who needs absolute value anyway?

__builtin__.abs = abs
# *Every* module now sees your terrible monkeypatch, on your own head be it
ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • @user2357112: That edit wasn't actually necessary; `__builtins__` (with `s`) is a weird auto-imported CPython implementation detail that provides the same attributes as the official (but not auto-imported) `__builtin__` (without `s`) provides. I'll grant the distinction is confusing, so it doesn't hurt anything to use `__builtin__` consistently. – ShadowRanger Sep 17 '18 at 20:30
  • 2
    The behavior of `__builtins__` in a module is subtly different from its behavior in interactive mode; specifically, inside a module, `__builtins__` will usually be the `__builtin__` module's `__dict__` rather than the module itself. Using `__builtin__`/`builtins` instead of `__builtins__` avoids this quirk. – user2357112 Sep 17 '18 at 20:35
  • @user2357112: True. Forgot about that bit of weirdness. I withdraw my (extremely minor) objection. :-) – ShadowRanger Sep 17 '18 at 20:37
  • I take the *print* example... and it is a special case, sigh! Now it is clear why on old python 2.x I cannot patch it. – Massimo Sep 19 '18 at 15:55