19

Couldn't find a way to phrase the title better, feel free to correct.

I'm pretty new to Python, currently experimenting with the language.. I've noticed that all built-ins types cannot be extended with other members.. I'd like for example to add an each method to the list type, but that would be impossible. I realize that it's designed that way for efficiency reasons, and that most of the built-ins types are implemented in C.

Well, one why I found to override this behavior is be defining a new class, which extends list but otherwise does nothing. Then I can assign the variable list to that new class, and each time I would like to instantiate a new list, I'd use the list constructor, like it would have been used to create the original list type.

class MyList(list):
    def each(self, func):
       for item in self:
           func(item)
list = MyList

my_list = list((1,2,3,4))
my_list.each(lambda x: print(x))

Output:

1
2
3
4

The idea can be generalize of course by defining a method that gets a built it type and returns a class that extends that type. Further more, the original list variable can be saved in another variable to keep an access to it.

Only problem I'm facing right now, is that when you instantiate a list by its literal form, (i.e. [1,2,3,4]), it will still use the original list constructor (or does it?). Is there a way to override this behavior? If the answer is no, do you know of some other way of enabling the user to extend the built-ins types? (just like javascript allows extending built-ins prototypes).

I find this limitation of built-ins (being unable to add members to them) one of Python's drawbacks, making it inconsistent with other user-defined types... Overall I really love the language, and I really don't understand why this limitation is REALLY necessary.

wjandrea
  • 28,235
  • 9
  • 60
  • 81
rboy
  • 750
  • 9
  • 20
  • 12
    [Zen of Python](http://www.python.org/dev/peps/pep-0020/) - "Explicit is better than implicit." Basically, it's probably not going to be obvious to other people working on your code that you have overridden such a basic feature of the language, so it's better to be explicit when doing things like that. – SethMMorton Sep 29 '13 at 20:57
  • 1
    No, you cannot override any built-in literal syntax; literals exist for convenience, there is no literal syntax for custom types *at all*. That does add clarity; you now always *know* what type literal syntax produces. – Martijn Pieters Sep 29 '13 at 20:59
  • 1
    As a side note, I find this limitation of python a feature, not a drawback. I can easily read someone else's code and not have to worry that (for example) `{}` won't create a `dict`. – SethMMorton Sep 29 '13 at 21:02
  • 3
    While there is a [module](http://clarete.github.io/forbiddenfruit/) designed to allow patching built-in types, it's a bad idea. – user2357112 Sep 29 '13 at 21:02
  • Well, Python does allow a lot of flexibility - for one, it allows to override the `list` variable, like I showed in my example.. This can also make it hard to other people to understand what I mean by `list((1,2,3))`... I think overriding the syntax literal would have the same weight on that Zen of Python rule. But since that's impossible, I wonder why the language does not allow adding members to builtins (what's the real reason behind it). – rboy Sep 29 '13 at 21:09
  • 2
    @RB14 Because Python is already dynamically typed, and if the modules you were importing were able to override and add members to built in types willy nilly, you would have no warning, and would be left scratching your head at unexpected behavior. There is no way for a person who has added a method to `list` in a module to advertise this to anyone who uses this code in the future. I don't see what the problem would be with using your own type which can take a plain old list or generator as a constructor argument. You can use something like `mylist([1,2,3,4])` while being concise. – Asad Saeeduddin Sep 29 '13 at 21:14
  • 2
    The real reason is that Python isn't Ruby (or any other language). – SethMMorton Sep 29 '13 at 21:14
  • 1
    Overriding the behaviour of properties of builtin objects goes against the zen of python since it makes things behave different from what you would expect. If you want to do these type of things though, I would recommend just using Ruby instead. – Wolph Sep 29 '13 at 21:17
  • 1
    You could rewrite the AST using the `ast` module when loading your program. – filmor Sep 29 '13 at 21:22
  • 1
    Another Zen of Python point: "There should be one—and preferably only one—obvious way to do it." There is already a way to do what `each` does: list comprehensions. `[function(x) for x in my_list]` does exactly what you want `each` do and will be universally understood by all python programmers. – HardlyKnowEm Sep 29 '13 at 21:28
  • 1
    @mlefavor Unfortunately, there is also the `map` function: `map(function, my_list)` does the same thing, But I TOTALLY agree with your sentiment and wish I didn't just point this out because there SHOULD be one—and preferably only one—obvious way to do it. – SethMMorton Sep 29 '13 at 21:30
  • 1
    @SethMMorton @miefavor Actually, `for x in my_list: print x` Is the one way to do it...no extraneous output list generated :^) – Mark Tolonen Sep 29 '13 at 21:36
  • @MarkTolonen I was not thinking about the context of this question anymore, so in my head (and probably miefavor's) `function` was anything but print. In the case of print, however, I agree with you. – SethMMorton Sep 29 '13 at 21:38
  • @mlefavor I don't think your solution is quite the same as my `each` function. You solution actually builds a list (even if it's a temp list), and can use a lot of memory just for invoking a function.. My solution just iterates over the elements and doesnt construct the list. as for the `map` function, in Python3 it actually an object and unless it's iterated (or wrapped in `list` or something), this function does effectively nothing. All that said, there is the possibility that Python makes optimizations to lists not assigned to variables, so it won't use space for the whole list. – rboy Sep 29 '13 at 21:38
  • @mlefavor Anyway, the question wasn't about the `each` function, was it? It was just an example... – rboy Sep 29 '13 at 21:39
  • @SethMMorton They were debating removing `map` and a bunch of other function composition tools for the very reason you mentioned in 3, but decided on keeping it. I don't understand why they decided to keep the redundant `map`, yet removed a useful tool like `reduce`. – Asad Saeeduddin Sep 30 '13 at 13:08
  • Totally beside the point, but `lambda x: print(x)` could be simplified to just `print`. – wjandrea Jan 26 '20 at 17:50

1 Answers1

35

This is a conscientious choice from Python.

Firstly, with regards to patching inbuilt type, this is primarily a design decision and only secondarily an optimization. I have learnt from much lurking on the Python Mailing List that monkey patching on builtin types, although enjoyable for small scripts, serves no good purpose in anything larger.

Libraries, for one, make certain assumptions about types. If it were encouraged to extend default types, many libraries would end up fighting each other. It would also discourage making new types – a deque is a deque, an ordered set is an ordered set, a dictionary is a dictionary and that should be that.

Literal syntax is a particularly important point. If you cannot guarantee that [1, 2, 3] is a list, what can you guarantee? If people could change those behaviours it would have such global impacts as to destroy the stability of a lot of code. There is a reason goto and global variables are discouraged.

There is one particular hack that I am fond of, though. When you see r"hello", this seems to be an extended literal form.

So why not r[1, 2, 3]?

class ListPrefixer:
    def __init__(self, typ):
        self.typ = typ

    def __getitem__(self, args):
        return self.typ(args)

class MyList(list):
    def each(self, func):
        return MyList(func(x) for x in self)

e = ListPrefixer(MyList)

e[1, 2, 3, 4].each(lambda x: x**2)
#>>> [1, 4, 9, 16]

Finally, if you really want to do deep AST hacks, check out MacroPy.

Veedrac
  • 58,273
  • 15
  • 112
  • 169
  • 6
    +1 for "If you cannot guarantee that `[1, 2, 3]` is a list, what can you guarantee?" And regarding your example: you're nuts! :) – Benjamin Hodgson Sep 29 '13 at 21:25
  • Excellent explanation of why it is "bad", and excellent proposed workaround! – SethMMorton Sep 29 '13 at 21:26
  • That's a great answer. You clarified some things for me.. What you said about assumptions made by libraries make perfect sense. Love your example(!), though it's a workaround for only a particular case about lists... Thanks for the reference for MacroPy, I'll definitely take a look. – rboy Sep 29 '13 at 21:57