27

I have some trouble with loading a pickled file in a module that is different from the module where I pickled the file. I am aware of the following thread: Unable to load files using pickle and multipile modules. I've tried the proposed solution of importing the class into the module where I am unpickling my file, but it keeps giving me the same error: AttributeError: Can't get attribute 'Document' on <module '__main__' from ''>

The basic structure of what I am trying to do:

Util file that pickles and unpickles objects, utils.py:

import pickle

def save_document(doc):

    from class_def import Document

    write_file = open(file_path, 'wb')
    pickle.dump(doc, write_file)

def load_document(file_path):
    from class_def import Document

    doc_file = open(file_path, 'rb')
    return pickle.load(doc_file)

File where Document object is defined and the save util method is called, class_def.py:

import utils

class Document(object):
    data = ""

if __name__ == '__main__':
    doc = Document()
    utils.save_document(doc)

File where the load util method is called, process.py:

import utils

if __name__ == '__main__':
     utils.load_document(file_path)

Running process.py gives the mentioned AttributeError. If I import the class_def.py file into process.py and run its main method as mentioned in the original thread it works, but I want to be able to run these two modules separately, since the class_def file is a preprocessing step that takes quite some time. How could I solve this?

Community
  • 1
  • 1
lmartens
  • 1,432
  • 2
  • 12
  • 19
  • 1
    Doesn't answer your question, but you should close the file handlers after you've opened them, or simply use [`with`](http://effbot.org/zone/python-with-statement.htm). – Itay Oct 27 '16 at 14:49
  • when a file is being run as `__main__` anything that are defined in it are pickled as members of the `__main__` module, so if you are then loading into a different file it fails. You need to only pickle things from modules other then the `__main__` the quickest fix is `from import *` but I do not recommend it in production code. – Tadhg McDonald-Jensen Oct 27 '16 at 15:29
  • possible duplicate of [namespace on python pickle](http://stackoverflow.com/q/7928450/5827215) but I think there might be a better thread to mark as dup – Tadhg McDonald-Jensen Oct 27 '16 at 15:33
  • check this thread https://stackoverflow.com/a/51397373/7429675, you can do something like this – Pankaj Saini Aug 12 '19 at 08:52

2 Answers2

27

in your class_def.py file you have this code:

if __name__ == '__main__':
    doc = Document()
    utils.save_document(doc)

This means that doc will be a __main__.Document object, so when it is pickled it is expecting to be able to get a Document class from the main module, to fix this you need to use the definition of Document from a module called class_def meaning you would add an import here:

(in general you can just do from <own module name> import * right inside the if __name__ == "__main__")

if __name__ == '__main__':
    from class_def import Document 
    # ^ so that it is using the Document class defined under the class_def module
    doc = Document()
    utils.save_document(doc)

that way it will need to run the class_def.py file twice, once as __main__ and once as class_def but it does mean that the data will be pickled as a class_def.Document object so loading it will retrieve the class from the correct place. Otherwise if you have a way of constructing one document object from another you can do something like this in utils.py:

def save_document(doc):
    if doc.__class__.__module__ == "__main__":
        from class_def import Document #get the class from the reference-able module
        doc = Document(doc) #convert it to the class we are able to use


    write_file = open(file_path, 'wb')
    pickle.dump(doc, write_file)

Although usually I'd prefer the first way.

Tadhg McDonald-Jensen
  • 20,699
  • 5
  • 35
  • 59
2

I had a similar problem and only just realized the differences between our implementations.

Your file structure:

  • util.py
    • define pickle functions
  • class_def.py
    • import util
    • define class
    • make instance
    • call save pickle
  • process.py
    • import util
    • load pickle

My mistake (using your file names) was first:

  • util_and_class.py
    • define class
    • define pickle funcs
    • make instance
    • call save pickle
  • process.py
    • import util_and_class
    • call load pickle << ERROR

What solved my pickle import problem:

  • util_and_class.py
    • define class
    • define pickle funcs
  • pickle_init.py
    • import util_and_class
    • make instance
    • call save pickle
  • process.py
    • call load pickle

This had the welcomed side effect that I didn't need to import the util_and_class file as it's baked into the pickle file. Calling the instance and saving the pickle in a separate file resolved the __name__ issue of "loading a pickled file in a module that is different from the module where I pickled the file."

xtian
  • 2,765
  • 7
  • 38
  • 65