4

Within the functools pacakge in Python3, there is a lru_cache() decorator that will memo-ize your function calls.

Is there a way for me to dump out this cache into a file and then load the file back into in-memory later?

I couldn't find this feature in the functools documentation. What would be the recommended way to achieve the above requirements, preferably with the solution involving only Python?

Rao
  • 20,781
  • 11
  • 57
  • 77
Spinor8
  • 1,587
  • 4
  • 21
  • 48

2 Answers2

3

I don't know a standart way to resolve this problem. But you can write your annotation like this:

def diskcached(cachefile, saveafter=1):
    def cacheondisk(fn):
        try:
            with open(cachefile, 'rb') as f:
                cache = pickle.load(f)
        except:
            cache = {}
        unsaved = [0]

        @wraps(fn)
        def usingcache(*args, **kwargs):
            try:
                key = hash((args, kwargs))
            except TypeError:
                key = repr((args, kwargs))
            try:
                ret = cache[key]
            except KeyError:
                ret = cache[key] = fn(*args, **kwargs)
                unsaved[0] += 1
                if unsaved[0] >= saveafter:
                    with open(cachefile, 'wb') as f:
                        pickle.dump(cache, f)
                    unsaved[0] = 0
            return ret

        return usingcache

    return cacheondisk

and use with 

@diskcached("filename_to_save")
def your_function():
    ...
MAXbrainRUS
  • 31
  • 3
2

Here is a different solution using the third party package joblib (pip install joblib), which has served me very well:

from joblib import Memory
memory = Memory("/usr/src/app/no_sync/tmp/", verbose=0)

@memory.cache
def args_and_results_of_this_function_are_cached_to_disk(a,b):
    return a + b
tyrex
  • 8,208
  • 12
  • 43
  • 50
  • 1
    Just tried it out and seems to work fine. However, caching across jupyter notebook sessions does not work currently. – gebbissimo Mar 04 '23 at 22:36
  • 1
    @gebbissimo interesting re Jupyter notebooks. I would try putting the function to be cached into a file outside the Juptyer notebook and import it in 2 different Jupyter notebooks or sessions. This may make it work across Jupyter notebook sessions – tyrex Mar 06 '23 at 10:38