Hashing a pandas dataframe for calculated column caching

Question

I am using composition method to create a class with a contained pandas dataframe as shown below. I am creating a derived property by doing some operation on the base columns.

import numpy as np
import pandas as pd

class myclass:
    def __init__(self, *args, **kwargs):
        self.df = pd.DataFrame(*args, **kwargs)
    @property
    def derived(self):
        return self.df.sum(axis=1)

myobj = myclass(np.random.randint(100, size=(100,6)))
d = mc.derived

The calculation of derived is an expensive step and hence I would like to cache this function. I want to use functools.lru_cache for the same. However, it requires that the original object be hashed. I tried creating a __hash__ function for the object as detailed in this answer https://stackoverflow.com/a/47800021/3679377.

Now I run in to a new problem where the hashing function is an expensive step!. Is there any way to get around this problem? Or have I reached a dead end?

Is there any better way to check if a dataframe has been modified and if not, keep returning the same hash?

Answer 1

If hashing doesn't work for you can try to take advantage of the internal state of your class.

Cache one method

Use a class attribute as a cache: on first call of the method, store the result into this attribute, and retrieve it on subsequent call.

import pandas as pd

class MyClass:
    def __init__(self, *args, **kwargs):
        self._df = pd.DataFrame(*args, **kwargs)
        self._cached_value = None

    @property
    def df(self):
        return self._df

    @df.setter
    def df(self, value):
        self._cached_value = None
        self._df = value

    @property
    def derived(self):
        if self._cached_value is None:
            self._cached_value = self._df.sum(axis=1)
        return self._cached_value

cl = MyClass()
cl.derived  # compute
cl.derived  # return cached value

cl.df = my_new_df_value  # cache is emptied
cl.derived  # compute

Cache several methods

You can then extend this principle to several methodes using a dict to store the result of each operation. You can use methods name as the keys to this dict (thanks to module inspect, see this response for an example).

import pandas as pd
import inspect

class MyClass:
    def __init__(self, *args, **kwargs):
        self.df = pd.DataFrame(*args, **kwargs)
        self._cached_values = {}

    @property
    def derived(self):
        method_name = self._get_method_name()
        if method_name not in self._cached_values:
            self._cached_value[method_name] = self.df.sum(axis=1)
        return self._cached_value[method_name]

    @property
    def derived_bis(self):
        method_name = self._get_method_name()
        if method_name not in self._cached_values:
            self._cached_value[method_name] = your_expensive_op
        return self._cached_value[method_name]

    def _get_method_name(self):
        return inspect.stack()[1][3]  # returns the name of this method's caller


cl = MyClass()
cl.derived  # compute  --> self._cached_value = {'derived': your_result}
cl.derived  # return cached value

cl.derived_bis # compute  --> self._cached_value = {'derived': your_result, 'derived_bis': your_other_result}
cl.derived_bis # return cached value

You can factorize the bodies of the two properties to respect the DRY principle, but be sure to modify _get_method_name accordingly.

Answer 2

If you know which methods are likely to update your df, you could override them in your custom class, and keep a flag. I'm not going into details here, but here is the basic principle:

import numpy as np
import pandas as pd

class myclass:
    def __init__(self, *args, **kwargs):
        self.df = pd.DataFrame(*args, **kwargs)
        self.derived_is_calculated = False
        
    @property
    def derived(self):
        if not self.derived_is_calculated:
            d = self.df.sum(axis=1)
            self.derived_is_calculated = True
            return d

    def update(self, other, **kwargs):
        """ Implements the normal update method, and sets a flag to track if df has changed """
        old_df = self.df.copy()  # Make a copy for comparison
        pd.DataFrame.update(self.df, other, **kwargs) # Call the base'update' method
        if not self.df.equals(old_df): # Compare before and after update
            self.derived_is_calculated = False
        
random_array = np.random.randint(100, size=(2,10))
myobj = myclass(random_array)

print(myobj.derived) # Prints the summed df
print(myobj.derived) # Prints None

myobj.update([1,2,3])
print(myobj.derived) # Prints the new summed df

There is probably a deeper method of DataFrame or pandas that is called on every change in the DataFrame content, I'll keep looking.

But you could setup a list of methods that your program will use, and make a decorator to do basically what I did in update and call it on each one of the listed methods...