3

Looking at this question, I realised that it is kind of awkward to use multiprocessing's Pool.map if you want is to run a list of functions in parallel:

from multiprocessing import Pool

def my_fun1(): return 1
def my_fun2(): return 2
def my_fun3(): return 3

with Pool(3) as p:
   one, two, three = p.map(lambda f: f(), [my_fun1, my_fun2, my_fun3])

I'm not saying it is exactly cryptic, but I think I expected some conventional name for this, even if only within functools or something, similarly to apply/call in JavaScript (yes, I know JavaScript didn't have lambdas at the time those functions were defined, and no, I'm not saying JavaScript is an exemplary programming language, just an example). In fact, I definitely think something like this should be present in operator, but (unless my eyes deceive me) it seems to be absent. I read that in the case of the identity function the resolution was to let people define their own trivial functions, and I understand it better in that case because there are a couple of different variations you may want, but this one feels like a missing bit to me.

EDIT: As pointed out in the comments, Python 2 used to have an apply function for this purpose.

jdehesa
  • 58,456
  • 7
  • 77
  • 121
  • 3
    Python used to have an `apply` builtin for this—but it was removed. And the reasoning was largely the same as for not adding identity: it’s trivial to write apply and all of its reasonable variations yourself, and rarely used, so if you want it, just write it yourself. – abarnert Mar 29 '18 at 16:15
  • `Pool.async_apply` would seem to be useful here. – chepner Mar 29 '18 at 16:22
  • @abarnert I see, I never knew (or forgot if I ever did) about `apply` in Python 2. I still think it would make sense in `operator` (like `reduce` was moved to [`functools`](https://docs.python.org/3/library/functools.html)), I mean, the same triviality argument applies to about anything in there, but who am I to question BDFL. Anyway, if you don't mind posting an answer with that I'll accept, I think it's the right explanation. – jdehesa Mar 29 '18 at 16:24
  • 1
    @jdehesa If you want me to dig up the full history and rationale for an answer, I will. But off the top of my head: Around 2.1, you couldn’t do perfect forwarding (because there was no keyword splat for lambdas), so the recommendation was to use apply. Then 2.2 fixed it, and Python has made sure to always allow perfect forwarding since then. So there were no recommended uses for apply, which opened a discussion that led to it being deprecated in 2.3, with a note showing how to write it (or variations on it) yourself. 3.0, removed it instead of moved to functools because nobody was using it. – abarnert Mar 29 '18 at 16:28
  • @chepner Yes, but `apply_async` takes a single function as parameter. You can call with each function, but if you want to extract the results to local function variables it can be weird with callbacks... – jdehesa Mar 29 '18 at 16:28
  • @abarnert Wow, thanks for the context, that's more than I could expect, there's no need for more digging! It sounds reasonable. I'll mark it as accepted answer if you post it. – jdehesa Mar 29 '18 at 16:31
  • OK, I dug up the easy-to-find links (2.7 docs, Python Regrets presentation) and didn't go digging into the python-3000 list archives or the archives of Guido's old blog. – abarnert Mar 29 '18 at 17:18

2 Answers2

5

First, let's look at the practical question.

For any Python from 2.3 on, you can trivially write not just your no-argument apply, but a perfect-forwarding apply, as a one-liner, as explained in the 2.x docs for apply:

The use of apply() is equivalent to function(*args, **keywords)

In other words:

def apply(function, *args, **keywords):
    return function(*args, **keywords)

… or, as an inline lambda:

lambda f, *a, **k: f(*a, **kw)

Of course the C implementation was a bit faster, but this is almost never relevant.1

If you're going to be using this more than once, I think defining the function out-of-line and reusing it by name is probably clearer, but the lamdba version is simple and obvious enough (even more so for your no-args use case) that I can't imagine anyone complaining about it.

Also, notice that this is actually more trivial than identity if you understand what you're doing, not less. With identity, it's ambiguous what you should return with multiple arguments (or keyword arguments), so you have to decide which behavior you want; with apple, there's only one obvious answer, and it's pretty much impossible to get wrong.


As for the history:

Python, like JavaScript, originally had no lambda. It's hard to dig up linkable docs for versions before 2.6, and hard to even find them before 2.3, but I think lambda was added in 1.5, and eventually reached the point where it could be used for perfect forwarding around 2.2. Before then, the docs recommended using apply for forwarding, but after that, the docs recommended using lambda in place of apply. In fact, there was no longer any recommended use of apply.

So in 2.3, the function was deprecated.2

During the Python-3000 discussions that led to 3.0, Guido suggested that all of the "functional programming" functions except maybe map and filter were unnecessary.3 Others made good cases for reduce and partial.4 But a big part of the case was that they're actually not trivial to write (in fully-general form), and easy to get wrong. That isn't true for apply. Also, people were able to find relevant uses of reduce and partial in real-world codebases, but the only uses of apply anyone could find were old pre-2.3 code. In fact, it was so rare that it wasn't even worth making the 2to3 tool transform calls to apply.

The final rationale for removing it was summarized in PEP 3100:

apply(): use f(*args, **kw) instead [2]

That footnote links to an essay by Guido called "Python Regrets", which is now a 404 link. The accompanying PowerPoint presentation is still available, however, or you can view an HTML flipbook of the presentation he wrote it for. But all it really says is the same one-liner, and IIRC, the only further discussion was "We already effectively got rid of it in 2.3."


1. In most idiomatic Python code that has to apply a function, the work inside that function is pretty heavy. In your case, of course, the overhead of calling the functions (pickling arguments and passing them over a pipe) is even heavier. The one case where it would matter is when you're doing "Haskell-style functional programming" instead of "Lisp-style"—that is, very few function definitions, and lots of functions made by transforming functions and composing the results. But that's already so slow (and stack-heavy) in Python that it's not a reasonable thing to do. (Flat use of decorators to apply a wrapper or three works great, but a potentially unbounded chain of wrappers will kill your performance.)

2. The formal deprecation mechanism didn't exist yet, so it was just moved to a "Non-essential Built-in Functions" section in the docs. But it was retroactively considered to be deprecated since 2.3, as you can see in the 2.7 docs.

3. Guido originally wanted to get rid of even them; the argument was that list comprehensions can do the same job better, as you can see in the "Regrets" flipbook. But promoting itertools.imap in place of map means it could be made lazy, like the new zip, and therefore better than comprehensions. I'm not sure why Guido didn't just make the same argument with generator expressions.

4. I'm not sure Guido himself was ever convinced for reduce, but the core devs as a whole were.

abarnert
  • 354,177
  • 51
  • 601
  • 671
  • Awesome, this is encyclopedic. I wasn't aware there was such "animosity" against higher-order functions! I'm glad they remained, I still find more idiomatic using `map` in some cases (less so `filter`), although I see how it may go against the "one obvious way to do it" zen. – jdehesa Mar 29 '18 at 17:42
  • 1
    @jdehesa Guido's position is often overstated. I think the best way to put it is not that he hates HOFs, but that he thinks an ideal language should have syntax that makes the commonly used HOFs unnecessary. And once you discover list comprehensions (and in Haskell, where HOFs are everything), it's not hard to believe that ideal is reachable. – abarnert Mar 29 '18 at 17:48
1

It sort of is in operator if you do one line of extra work:

>>> def foo():
...     print 'hi'
... 
>>> from operator import methodcaller
>>> call = methodcaller('__call__')
>>> call(foo)
hi

Of course, call = lambda f: f() is only one line as well...

timgeb
  • 76,762
  • 20
  • 123
  • 145
  • 1
    Right, not sure if one option is better than the other, but it is a nice alternative to the `lambda`. – jdehesa Mar 29 '18 at 16:40