Python - using 'partial' for late-binding problems not amenable to a lambda - costs vs benefits?

Question

Note: I'm asking if there's One Pythonic Way to do this (using default args seems less Pythonic than using partial) and if there are significant limitations to either method (the "cost" - I wouldn't expect timing to differ significantly but perhaps there are other limitations I'm not seeing that tilt the balance towards one methodology versus the other).

I'm trying to understand the costs of using 'partial' in late-binding situations where a lambda is not feasible. I've created some example code based on this guide to exemplify this.

The following doesn't work as intended due to late binding:

def create_thingies():
    thingies = []
    for i in range(1,6):
        def thingy(x):
            print("Some output", i)
            return i ** (x * i)
        thingies.append(thingy)
    return thingies

results=[]
for thingy in create_thingies():
    results.append(thingy(2))
print(results)

Output:

Some output 5
Some output 5
Some output 5
Some output 5
Some output 5
[9765625, 9765625, 9765625, 9765625, 9765625]

Using 'partial' we avoid that problem, but at what cost?

from functools import partial
def create_thingies():
    thingies = []
    for i in range(1,6):
        def thingy(i, x):
            print("Some output", i)
            return i ** (x * i)
        thingies.append(partial(thingy, i))
    return thingies

results=[]
for thingy in create_thingies():
    results.append(thingy(2))
print(results)

Output:

Some output 1
Some output 2
Some output 3
Some output 4
Some output 5
[1, 16, 729, 65536, 9765625]

I've seen much discussion about lambda vs partial here, but in cases where a lambda wouldn't work well (a very complex function) if at all (function with more than expressions) is partial the way to go or is there a better way short of coercing this into a lambda expression?

kindall's answer is how we normally do it. Are you looking for that or to understand the use of `partial` in this case? — thefourtheye, May 01 '14 at 01:41
I added a comment to their answer. I've seen that lambdas are preferred to partials (Guido), but when lambdas aren't usable is using an argument side-effect preferable to an explicit partial call? And if so, why? — MartyMacGyver, May 01 '14 at 01:59
Wait, lambdas are not preferred at all. Python community likes to move away from lambdas as much as possible. Secondly, we use default parameterized function and that is perfectly fine — thefourtheye, May 01 '14 at 02:20
It's not very clear what you're asking for in this question. Do you want somebody to do a performance comparison of `partial` versus other solutions? Do you want the "most Pythonic" solution, or the simplest to write? How can we assess what answer could "solve" your question? Stack Overflow isn't intended as a place for open ended discussions. — Blckknght, May 01 '14 at 03:02
@thefourtheye That depends on who you ask... and when (http://stackoverflow.com/questions/3252228/python-why-is-functools-partial-necessary) — MartyMacGyver, May 01 '14 at 16:44
@Blckknght I was primarily looking for the Pythonic way to do this (to me using side-effects isn't Pythonic so I'd think partial is the way to go, but as above there is some debate here and elsewhere on the matter and my question limits this to default args versus partial). Then I wondered, is there a significant difference between the two methodologies? Not so much timing but perhaps other limitations. — MartyMacGyver, May 01 '14 at 16:47

chepner · Answer 1 · 2014-05-01T03:12:59.703

Using partial, there is no need to define thingy once for each value of i, since thingy does not use any free/global variables, but just its parameters.

from functools import partial

def thingy(i, x):
    print("Some output", i)
    return i ** (x * i)

thingies = [partial(thingy, i) for i in range(1,6)]
results = [th(2) for th in thingies]
print(results)

As for cost, you should profile to see if the performance is acceptable.

Here's a quick test to compare 3 options:

import timeit

# The fastest: define a function using a default parameter value
print timeit.timeit('results = [ th(2) for th in create_thingies()]', '''
def create_thingies():
    thingies = []
    for i in range(1,6):
        def thingy(x,i=i):
            #print("Some output", i)
            return i ** (x * i)
        thingies.append(thingy)
    return thingies
''')

# The slowest, but IMO the easiest to read.
print timeit.timeit('results = [ th(2) for th in create_thingies()]', '''
def create_thingies():
    from functools import partial
    def thingy(i,x):
        #print("Some output", i)
        return i ** (x * i)
    return [partial(thingy, i) for i in range(1,6)]
''')

# Only a little slower than the first
print timeit.timeit('results = [ th(2) for th in create_thingies()]', '''
def create_thingies():
    def make_thingy(i):
        def thingy(x):
            #print("Some output", i)
            return i ** (x * i)
        return thingy
    thingies = [make_thingy(i) for i in range(1,6)]
    return thingies
''')

Question updated - I wouldn't expect timing to differ but wondered if there are other factors involved in using partials vs default arguments. — MartyMacGyver, May 01 '14 at 16:53

score 3 · Answer 2 · answered May 01 '14 at 01:35

3

Use a default argument nalue to bind the value at function creation:

def thingy(x, i=i):
    print("Some output", i)
    return i ** (x * i)

answered May 01 '14 at 01:35

kindall

178,883
35
278
309

I understand that this is more of a hack based on a side-effect of how default arguments work though... is doing this really Pythonic, and how is it preferable to 'partial'? – MartyMacGyver May 01 '14 at 01:57
1

I personally like it because it doesn't require an import and makes it obvious what arguments are being bound right in the signature. – kindall May 05 '14 at 03:30

score 3 · Answer 3 · answered May 01 '14 at 03:08

There are a few ways to do early binding. Some of the most popular are default args, partial, and maker functions. At least on my machine with my version of python, they all take approximately the same amount of time.

Here are examples of how to do each of the three:

import time
from functools import partial
from contextlib import contextmanager

@contextmanager
def timer(what):
    t1 = time.time()
    yield
    print "%-30s: %5d millis" % (what, (time.time() - t1) * 1e3)

N = 5000

print
with timer('create bound'):
    thingies = []
    for i in xrange(N):
        def thingy(x, i=i):
            return i ** (x * i)
        thingies.append(thingy)
with timer('eval bound'):
    for t in thingies:
        t(2)

with timer('create partial'):
    def thingy(i, x):
        return i ** (x * i)
    thingies = [partial(thingy, i) for i in xrange(N)]
with timer('eval partial'):
    for t in thingies:
        t(2)

with timer('create maker'):
    def make_thingy(i):
        def thingy(x):
            return i ** (x * i)
        return thingy
    thingies = [make_thingy(i) for i in xrange(N)]
with timer('eval maker'):
    for t in thingies:
        t(2)

and here are the times I observe (Python 2.7.6 + Windows + Haswell):

create bound                  :     5 millis
eval bound                    :  1861 millis
create partial                :     2 millis
eval partial                  :  1832 millis
create maker                  :     2 millis
eval maker                    :  1829 millis

Note that creating the bound methods is more expensive, but the calling overhead is pretty close for all 3 versions.

I generally use a mix of partials and maker functions, depending on which is clearest for a given bit of code.

It's interesting to see the timings but I was thinking more of the other costs (or benefits) of using partial vs default arguments. The maker function idea is novel... I hadn't considered that. — MartyMacGyver, May 01 '14 at 16:54
@MartyMacGyver The main costs are: time, memory, and readability. All three should have similar memory footprints unless you're creating a very large number of tiny functions (in which case your application is likely very slow). Between these three options, there are application-specific readability tradeoffs. partial is commonly the best, but not always. Default args allow users to override the default later (sometimes good, sometimes bad). Etc. — Mr Fooz, Sep 25 '18 at 14:51

Python - using 'partial' for late-binding problems not amenable to a lambda - costs vs benefits?

3 Answers3