I've stumbled upon something strange. Basically, I use a library that takes the urllib.parse.urljoin function and wraps it in functools.lru_cache. But it applies the lru_cache "decorator" on it in-line - dynamically, like this:
from urllib.parse import urljoin
lru_cache()(urljoin)
It works fine, until I want to go for multiprocessing (concurrent.futures), which uses pickling. Then I get a pickling error.
This is a reproducible example I made myself:
import pickle
from functools import lru_cache
# Working pickle
@lru_cache()
def foo(n):
return n**2
a = pickle.dumps(foo)
b = pickle.loads(a)
print(b(3))
# Not working pickle
def bar(n):
return n**2
x = lru_cache()(bar)
a = pickle.dumps(x)
b = pickle.loads(a)
print(b(3))
Output:
C:\Users\dabljues\Desktop> python .\cache_example.py
9
Traceback (most recent call last):
File "C:\Users\dabljues\Desktop\cache_example.py", line 23, in <module>
a = pickle.dumps(x)
_pickle.PicklingError: Can't pickle <functools._lru_cache_wrapper object at 0x00000242624C2CF0>: it's not the same object as __main__.bar
So, as you can see, pickling works when the lru_cache is applied in a decorator style, it doesn't when it's applied inline on a function.
My question is: is there a way around it? I mean foo and x are both functools._lru_cache_wrappers, so why doesn't it work?
@Edit
Those are the __module__s and __qualname__s of bar with and without lru_cache:
import pickle
from functools import lru_cache
def bar(n):
return n**2
x = bar
y = lru_cache()(bar)
print(x.__module__, x.__qualname__)
print(y.__module__, y.__qualname__)
__main__ bar
__main__ bar
