0

Introduction to the problem

Hi, I have recently switched to Python programming language from Mathematica because I would like to make my code portable and more powerful. I studied the Functional Programming HOWTO guide and I started playing around with higher-order functions.

What I find confusing for a newcomer on the functional paradigm of the Python language is the default behavior, i.e. the standard execution, of higher-order functions. For example when you apply map() over a sequence you get back a map object (heck see the comments below):

odd = lambda x : x%2!=0
lis = [1, 6, 2, 5, 9, 4]
map(odd, lis)

Out[171]: <map at 0x19e6a228e48>

Mathematica users would expect to "thread" odd() over a list and the result of the evaluation would be a list of booleans. In python you have to materialize the result using the list() constructor e.g.:

list(map(odd, [1, 6, 2, 5, 9, 4]))

Out[172]: [True, False, False, True, True, False]

What I am missing

One of the things I am missing in Python is a list-able attribute for thread-able functions. Indeed this is a core feature in Wolfram Mathematica language. But the beautiful thing in Python is that everything is an object (everything is an expression in Wolfram Language) including functions therefore I can change how function objects behave by passing a keyword argument to indicate whether I want the function to return a generator/iterator or the full materialized result.

Specifications for a full answer

So this is the question to ask here for advanced core developers of the Python Language. Continuing the example above, odd() is a function that takes one argument, if PyFunctionObject had, let's say, a materialize and listable attribute I would expect to write

odd.listable = True
odd.materialize = True
odd(1, 6, 2, 5, 9, 4)

Out[172]: [True, False, False, True, True, False]

odd(6)

Out[173]: False

Or switch to the default behavior you get now when you map() ...

odd.listable = True
odd.materialize = False
odd(1, 6, 2, 5, 9, 4)

Out[31]: <generator object Listable.__call__.<locals>.<genexpr> at 0x000001F3BBF1CC50>

References

I have searched stackoverflow for similar questions and the closest I have found is this one: Automatically use list comprehension/map() recursion if a function is given a list. The answer of David Robinson is based on decorators. Back in 1999 Michael Vanier posted also this answer here which is a class based solution of this problem.

My question is slightly different because I am asking how you can tweak the function object at a low level so that you get the desirable behavior I wrote about. I am also arguing here that this feature will make functional programming in Python easier for newcomers and a lot more fun, For a start, they do not need to learn about generators and iterators. If there is already such discussion in the road map to Python please let me know.

Athanassios
  • 206
  • 1
  • 10
  • 1
    I'm not sure I understand what you're asking; something like a syntax change to the language? Note that `map` [isn't even really liked by the creator of Python](https://www.artima.com/weblogs/viewpost.jsp?thread=98196). Python would have to work hard to be any easier than it already is (I'm not sure what that would look like) and it's not just for mathematical work so the features have to support a huge range of use cases. I think you're approaching this with the wrong mindset; you're probably not approaching your problem in a pythonic way. – roganjosh Aug 25 '17 at 17:54
  • 1
    "For example when you apply map() over a sequence you get back a generator" No, you **do not**. You get a `map` object. – juanpa.arrivillaga Aug 25 '17 at 17:59
  • How about the [`vectorize`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.vectorize.html) in `numpy`? – stamaimer Aug 25 '17 at 18:04
  • @roganjosh yes you are right I think functional programming could become more fun and easier with some syntactic sugar. So in Mathematica if the function is listable you can do fun@lis or even lis // fun, i.e. postfix and prefix notation. Now I think in terms of Composability that makes it far more easier to test and write cleaner code – Athanassios Aug 25 '17 at 18:17
  • Ah, no, I disagree that this makes code easier to test. Python is not "complete" in terms of having all of this kind of syntactic sugar; it's a general-purpose language. However, it's supported by a _huge_ number of libraries that build on this foundation and provide basically exactly what you suggest so that projects that need such functionality have access to it. – roganjosh Aug 25 '17 at 18:20
  • @juanpa.arrivillaga ok a map object but when I read the Functional Programming how to it says (map() and filter() duplicate the features of generator expressions). So what is behind a map object ? In my opinion this is causing a confusion and I can full appreciate why Guido does NOT like having both list comprehensions and map at the same time. – Athanassios Aug 25 '17 at 18:21
  • A generator is a *language construct*. `map` objects are *iterators*. Generators were introduced to make writing iterators more succinct and streamlined. So `map` objects are simply objects that implement `__iter__` and `__next__`. Generators *are* iterators, but not all iterators are generators. Check out this [answer](https://stackoverflow.com/a/45685692/5014455) – juanpa.arrivillaga Aug 25 '17 at 18:24
  • Also, `import types` and see what `isinstance(map(lambda x:x, range(10)), types.GeneratorType)` gives you... – juanpa.arrivillaga Aug 25 '17 at 18:29
  • @juanpa.arrivillaga I got your point. Yes I understand the difference, it's about memory allocation, correct ? I was misled by the Functional Programming guide. Still, what I do not understand is why do I have to materialize the call of this iterator with the sequence to get the result ? In other sequence types all function calls, i.e. methods? return immediately the result – Athanassios Aug 25 '17 at 18:38
  • No, the difference is *not about memory allocation*. Again, *generators* are returned form generator functions (a language construct), which are a subset of iterators. But not all iterators are generators. Iterators **are not sequence types**, if you want to materialize any iterator, you generally call `list` or `tuple` or `set` or whatever you want. The `map` function that you call *is not an iterator*, it *returns an iterator*. – juanpa.arrivillaga Aug 25 '17 at 18:40

2 Answers2

2

There is already a perfectly Pythonic way to "materialize" lazy constructs. Wrap it in list(). The list constructor takes any sequence and converts it to a list.

>>> odd(1, 6, 2, 5, 9, 4)
<generator object odd at ...>

list(odd(1, 6, 2, 5, 9, 4))
[True, False, False, True, True, False]

The idea of setting "switches" on a function to change its behavior means that functions can no longer even possibly be "pure." It's neither Pythonic nor functional.

kindall
  • 178,883
  • 35
  • 278
  • 309
  • I fully agree with that but it may be sensible to explain why this would not be pythonic. – Ramon Aug 25 '17 at 17:56
  • Because it's usually something you'd do with an argument, if you were going to do it at all. – kindall Aug 25 '17 at 18:04
  • Hi Kindall, in Mathematica all these functions are immutable and listability is an attribute that you can set on a function. Add to this the generall impression that I got from Python is that it is a language with a lot of freedom and at the same time responsibility from experienced users to code at the right direction. That said I am not a core developer and obviously I cannot judge whether such a feature at a low level could be hazardous – Athanassios Aug 25 '17 at 18:07
  • @Athanassios I fundamentally agree with kindall, such a feature is neither Pythonic nor functional - there seems to be some sort of disconnect between the claim that in Mathematica, functions are immutable, and yet, you can set attributes on the function... – juanpa.arrivillaga Aug 25 '17 at 18:27
  • @juanpa.arrivillaga first thank you indeed both of you for the lessons in Python ;-) I am really trying to understand the issue here. It seems that there is a clear seperation between language constructs i.e. generators, iterators, functions and built-in sequence types, i.e. container objects correct ? – Athanassios Aug 25 '17 at 18:49
  • @Athanassios yes, these are all separate things, although, intimately related. [Here](http://nvie.com/posts/iterators-vs-generators/) is a decent summary, with good diagrams! **Edit**, one misleading thing, a *generator function* is **not** a generator, it is a function. A generator expression *is* a generator (or perhaps, it evaluates to a generator is more precise). However, the rest of that diagram is pretty good – juanpa.arrivillaga Aug 25 '17 at 18:51
  • @juanpa.arrivillaga ah this is a great diagram indeed, thank you. It should definitely go at the official Python documentation guide. – Athanassios Aug 25 '17 at 19:12
  • @kindall according to the way I have defined odd() in my question, i.e. as a lambda function that takes one argument if you do odd(1, 6, 2, 5, 9, 4) you will get a TypeError: () takes 1 positional argument but 6 were given. You have to do a list comprehension or map() over the sequence then materialize with list(). Moreover I have already highlighted this when I was writing my question. Nevertheless your comments are useful, thanks. – Athanassios Aug 25 '17 at 21:04
0

This answer does not fully cover my question because I am asking how to modify the behavior of the core function type (is it PyFunctionObject?). Nevertheless, I thought to share it with the rest of the users because I learned a lot going through this piece of code and it is the closest I could reach as an answer. It is based on an old post of Michael Vanier back in 1999. Here it goes:

class Listable(object):
    """
    Listable functions, are functions which automatically map themselves over a sequence.  
    This idea is borrowed from Mathematica.
    """

    def __init__(self, f, materialize=True):
        self.func  = f
        self.gen = not materialize

    def __call__(self, *args):
        # check the number of arguments
        if len(args)==1:
            # return a scalar
            return self.func(*args)
        elif self.gen:
            # return a generator 
            return (self.func(x) for x in args)
        else:
            # return an iterator
            return [self.func(x) for x in args]
Shreyash S Sarnayak
  • 2,309
  • 19
  • 23
Athanassios
  • 206
  • 1
  • 10
  • Ok, this seems like it's already covered by the [numpy](http://www.numpy.org/) library. You mention "threadable" in your question, but actually I don't think that it's the same as what you're expecting in Python due to the Global Interpreter Lock. Numpy will allow you to vectorize calculations in near-C-time. If you're working with numbers, you probably want to use that, not some construct like you've made here. – roganjosh Aug 25 '17 at 18:07
  • This would be fine if you want to provide a similar interface as what is available in Mathematica, but I definitely do not think it should become part of the core language. Python and Mathematica are not similar languages at all. But Python is versatile and extendable enough to provide this for those that want it. – juanpa.arrivillaga Aug 25 '17 at 18:31
  • Anyway, Python is *not meant to be a functional language*. At it's core, it is very very imperative. – juanpa.arrivillaga Aug 25 '17 at 18:34
  • Also, your listable class *returns a **list**, not an iterator*. Lists are *iterable* but they are **not iterators**. Try to see what the interpreter tells you when you do `next([1,2,3])` – juanpa.arrivillaga Aug 25 '17 at 18:36
  • @juanpa.arrivillaga would you call list an iterable sequence type then ? And by the way my listable class returns also a generator which is an iterator if materialize=False is that correct ;-) – Athanassios Aug 25 '17 at 18:59
  • @Athanassios yes, a list is an iterable sequence type! And yes, `Listable` returns a generator iterator when `materialize=False` – juanpa.arrivillaga Aug 25 '17 at 19:29