Class variable is not shared over all instances after pickling and unpickling

Question

I have a class ConfProvider that contains a class variable which holds all config that cannot be pickled. My other classes inherit from this class and they use the _conf class variable when they call the non-pickable objects. The idea is then to clean the config when pickling my classes so I do not get any errors because the class contains unpickable objects.

This is my ConfProvider class

class ConfProvider:
    _conf = None
    
    @staticmethod
    def set_conf(conf: dict) -> None:
        ConfProvider._conf = conf

    @staticmethod
    def clear_conf() -> None:
        ConfProvider._conf = {}

    @classmethod
    def print_conf(cls) -> None:
        cls._conf["console_logger"].info(ConfProvider._conf)

Next to this class I have 2 other classes that inherit from the ConfProvider. Class_1 contains objects of class_2.

class Class_1(ConfProvider):

    def __init__(self):
        # Do some init stuff
        self.list_class_2 = [Class_2(1, 2), Class_2(3, 4)]

    @Logging.debug_decorator
    def save(self, debug=False) -> None:
        config = self._conf
        self.clear_conf()

        with open(save_path, "wb") as f:
            cloudpickle.dump(self, f)

        # Reset non serializable attributes
        self.set_conf(config)

class Class_2(ConfProvider):

    def __init__(self, a, b):
        # Do some init stuff

I want to be able to pickle the Class_1 object I create that contains several objects of Class_2. For this, I use the save method as shown above in Class_1. Now this works perfectly since the class variable is shared amongst all objects. However, once I unpickle the class_1 object and add new class_2 objects to the list in class_1, the clear_conf method does only clear the conf variable from the objects that were included before pickling. The _conf of all new objects is not cleared. How is this possible as normally a class variable is shared over all instances?

EDIT:

After debugging a bit more, I found that before pickling Class_1._conf always contains the same content as Class_1().__class__._conf. However, once I pickle the class and unpickle it, the two are different. Class_1._conf has the values that were loaded when starting up, but Class_1().__class__._conf contains an empty dictionary.

Is this helpful? https://stackoverflow.com/questions/6760685/creating-a-singleton-in-python — alec_djinn, Jul 11 '23 at 09:45
This seems to give more info about the singleton pattern, however, the problem with me is that the behaviour seems to be different after I pickle and unpickle the object — Lennart, Jul 11 '23 at 11:05
That is odd. Pickling doesn't change the object. Also, I think the singleton is exactly what you want. You want to have `_conf` as a singleton. — alec_djinn, Jul 11 '23 at 11:29
Don't have the answer but maybe 2 tracks to follow : 1 - is it the same if the 2 staticmethods are redefined as classmethods ? 2 - this post seems interesting for your problem: https://stackoverflow.com/a/50100169/13339621 — matleg, Jul 11 '23 at 11:58
Yes, the problem remains the same. In the meantime, I found that ```cls._conf``` is not the same as the ```ConfProvider._conf```, which seems weird to me, but if I print this, I get different results. The cls was called from within a subclass of the ConfProvider class, but nevertheless I would expect this to be the same result. — Lennart, Jul 12 '23 at 06:48

score 0 · Accepted Answer · answered Aug 02 '23 at 14:04

Eventually I found the problem. I used the cloudpickle module, not the normal pickle module. Cloudpickle will pickle the objects by value instead of by reference. This way, the classes themselves were also pickled. When I unpickled my objects, the already loaded classes were overridden which caused that two ConfProvider classes were present, each with its own class variable.

I was able to solve it by overriding the ConfProvider class that was loaded with the ConfProvider class present in the pickle file. Afterwards, everything worked as expected and the class variable was shared.

Class variable is not shared over all instances after pickling and unpickling

1 Answers1