1

I need to create a bunch of partial functions that can be pickled. Empirically testing this, it seems to work fine:

import pickle
import functools

pickle.dumps(functools.partial(int, base=2))
# b'\x80\x03cfunctools\npartial\nq\x00cbuiltins\nint\nq\x01\x85q\x02Rq\x03(h\x01)}q\x04X\x04\x00\x00\x00baseq\x05K\x02sNtq\x06b.'

However, this is not stated explicitly in the documentation, and the given "roughly equivalent" function of functools.partial() is not picklable.

Can I assume that partial objects are picklable in any circumstances (no edge cases, independently of the C / Python implementation)?

martineau
  • 119,623
  • 25
  • 170
  • 301
Delgan
  • 18,571
  • 11
  • 90
  • 141
  • 1
    Did you look at the [actual implementation](https://github.com/python/cpython/blob/3.7/Lib/functools.py#L234)? – jonrsharpe Oct 13 '19 at 07:21
  • @jonrsharpe: That implementation gets replaced by the `from _functools import partial` below. – user2357112 Oct 13 '19 at 07:22
  • @user2357112 possibly, yes, but the point is still that the roughly equivalent implementation isn't the actual implementation. – jonrsharpe Oct 13 '19 at 07:23
  • @jonrsharpe I was mainly worried about differences between C and Python implementations. Should I consider the "picklability" an implementation detail? It seems not according, partial objects not being picklable has been considered as a bug: https://bugs.python.org/issue1398 and https://bugs.python.org/issue27137 Still, can't find changelog about this. – Delgan Oct 13 '19 at 07:32
  • Could you [edit] to reflect that? The roughly equivalent version is irrelevant in that case. – jonrsharpe Oct 13 '19 at 07:34
  • @jonrsharpe I removed the irrelevant code snippet. – Delgan Oct 13 '19 at 07:40
  • 1
    Have you read the pickle docs? There's a whole section on how to create your own pickleable classes. – Legorooj Oct 13 '19 at 07:54
  • 1
    https://stackoverflow.com/questions/14550577/pickling-wrapped-partial-functions Did you read the comments to the answer here? – gstukelj Oct 13 '19 at 08:13
  • If it helps, as far as I can tell, pickle doesn't care much about cross-implementation compatibility. So it seems safe to assume that `partial` objects can be always be pickled, but you might run into problems if you pickle one with CPython and try to unpickle it with PyPy. – Aran-Fey Oct 13 '19 at 08:17

1 Answers1

3

The code snippet in the documentation simulating the partial() function is not picklable as it uses nested function. However, the official functools.partial() is implemented using a class which is pickable: cpython/functools.py#L234

The official Python issue tracker contains several references for partial objects not beeing picklable, and this has been considered as a bug:

Also, there are unit tests to check picklability of partial objects: cpython/test_functools.py#L244

The changelog for Python 3.6 contains reference to the picklability of functools.partial objects:

bpo-27137: the pure Python fallback implementation of functools.partial now matches the behaviour of its accelerated C counterpart for subclassing, pickling and text representation purposes.

For these reasons, I think one can safely assume that partial picklability is not an implementation detail and can be relied upon.

Delgan
  • 18,571
  • 11
  • 90
  • 141
  • As I mentioned in the comments on the question, the thing in `functools.py` is not the actual implementation. It's an alternate implementation that gets replaced immediately. – user2357112 Oct 13 '19 at 09:30