0

I have a very simple scenario. I have 1 custom class (not a NN model) I am pickling with torch.save. When I try to load it it fails with error:

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/Users/brando/anaconda3/envs/metalearning/lib/python3.8/site-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/Users/brando/anaconda3/envs/metalearning/lib/python3.8/site-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
AttributeError: Can't get attribute 'DagDataPreparation' on <module '__main__' from '/Users/brando/ML4Coq/ml4coq-proj/data_lib/dag/dag_dataloader.py'>

but the class is basically empty (though it's in a different file DagDataPreparation:

class DagDataPreparation:

    def __init__(self, root):
        print('1')

    def create_everything(self):
        return 1

if I use dill as an argument for torch.load and torch.save it works. It also works if I simply import the class at the top of the file

from data_lib.dag.dataset_preparation import DagDataPreparation

I understand why the error might be happening, it can't find the definition of the class...but this is really weird because I am 200% I've done this type of thing (saving/loading arbitrary classes with torch) and I've never had this issue before.

What might be the issue? Does everyone else do experience the same bug in my scenario if you have two files one that loads the pickle file and it throws that error at you?

Charlie Parker
  • 5,884
  • 57
  • 198
  • 323

1 Answers1

0

Well if it worked in the past it is because when loading the pickled object your model definition was somehow accessible from the current scope otherwise It'll have crashed. This is not a bug it is how pickle works.

I guess you've already seen the pytorch suggestion on that issue which discouraged saving the entire model instead just the weights for the same problem you just mentioned, but just in case here the link

Also see this answer answer which tells that pickle works by saving reference to the class whereas dill can save class definition and thus can have access to it when loading.

Alka
  • 767
  • 1
  • 6
  • 13
  • what I don't understand is that I have saved with pickle `Counters()` from collections before and those don't need an import explicitly...why do my classes do? Plus I do have other classes I've saved with torch before with no issue but now it doesn't work. Somehow it stopped working like it used to work. – Charlie Parker Mar 18 '21 at 18:08
  • btw, Im also not pickling models. Not sure if that matters. – Charlie Parker Mar 18 '21 at 18:19