Pickle functools wrapper error: can't pickle functools.KeyWrapper objects

Question

I am trying to pickle a SortedListWithKey for which I am using cmp_to_key() from functools to convert a comparison function to a key function. However, the cmp_to_key() seems to make my object unpickable and I get the following error: TypeError: can't pickle functools.KeyWrapper objects

How can I fix it? This is a code sample to reproduce the error:

import pickle
from functools import cmp_to_key
from sortedcontainers import SortedListWithKey

def order_fun(a, b):
    if abs(a[0]-b[0]) < 1e-8:
        return 0
    elif a[0]-b[0] > 0:
        return 1
    else:
        return -1

pickle.loads(pickle.dumps(SortedListWithKey([[1,2], [3,4]], key=cmp_to_key(order_fun))))

Thanks!

Note: The pickling works fine without using the cmp_to_key() function, but I need it since my function is not a key function.

can generators be pickled? maybe try: `list(SortedListWithKey(...))` ? see: https://stackoverflow.com/questions/7180212/why-cant-generators-be-pickled#7180424 by using the `key` kwarg i think it turns it into a generator? — jmunsch, Dec 27 '17 at 00:40
A very old question, not every objects can be pickled. Try converting to some basic objects. — Sraw, Dec 27 '17 at 01:09
You can't pickle a function (of any kind). There's an module named [`dill`](https://pypi.python.org/pypi/dill/0.2.7.1) that ought to be able to do it, though. — martineau, Dec 27 '17 at 01:45

score 2 · Accepted Answer · answered Dec 27 '17 at 01:43

The problem seems to be that cmp_tp_key is now written in C, and the class it returns is neither pickleable nor subclassable. However, the original pure python version is still maintained in the source, and is very simple. When this is used with your example, it works correctly. Of course, the obvious downside is that the pure python version is slower - but the difference is not huge.

Here is a working version of your example:

import pickle
from sortedcontainers import SortedListWithKey

def order_fun(a, b):
    if abs(a[0]-b[0]) < 1e-8:
        return 0
    elif a[0]-b[0] > 0:
        return 1
    else:
        return -1

class KeyFunc(object):
    __slots__ = ['obj']
    def __init__(self, obj):
        self.obj = obj
    def __lt__(self, other):
        return order_fun(self.obj, other.obj) < 0
    def __gt__(self, other):
        return order_fun(self.obj, other.obj) > 0
    def __eq__(self, other):
        return order_fun(self.obj, other.obj) == 0
    def __le__(self, other):
        return order_fun(self.obj, other.obj) <= 0
    def __ge__(self, other):
        return order_fun(self.obj, other.obj) >= 0
    __hash__ = None

sl = SortedListWithKey([[1,2], [3,4]], key=KeyFunc)

print(sl)

print(pickle.loads(pickle.dumps(sl)))

Output:

SortedListWithKey([[1, 2], [3, 4]], key=<class '__main__.KeyFunc'>)
SortedListWithKey([[1, 2], [3, 4]], key=<class '__main__.KeyFunc'>)

Pickle functools wrapper error: can't pickle functools.KeyWrapper objects

1 Answers1