4

I am trying to pickle a dynamically generated class as a factory for an alternative class. Something like the following:

import sys, pickle

class BC(object):
    pass

C = type("NewClassName", (BC,), {})

pickle.dump(C, sys.stdout)

This leads to the following error:

pickle.PicklingError: Can't pickle <class '__main__.NewClassName'>: it's not found as __main__.NewClassName

For pickling an object of a dynamically generated class, you can define an __reduce__ method, but is there a way to achieve this only for a class definition.

I don't want to use BC directly, because I only need it as a factory for new classes.

otus
  • 5,572
  • 1
  • 34
  • 48
eatdas
  • 149
  • 13
  • Why would you pickle a dynamically generated class? Can't you just simply serialize parameters you used to create that class and recreate it somewhere else? I mean pickling a class is metaprogramming taken a bit too far imho. And I can't think of a real life example. – freakish Jun 23 '14 at 11:15

3 Answers3

2

Try the following:

C = type("C", (BC,), {})

The class must be a module level variable, with the same name as the type name.

However, pickling a class dynamically generated class like this will not work (see the answer from @otus).

The best solution I can think of, is to pickle the arguments to type, and then recreate the class again when you unpickle it.

Pickle:

import sys, pickle

class BC(object):
    pass

args = ("NewClassName", (BC,), {})
C = type(*args)
C._pickle_args = args

pickle.dump(C._pickle_args, sys.stdout)

Unpickle:

type_args = pickle.loads("<pickled string">)
C = type(*args)
Jamie Cockburn
  • 7,379
  • 1
  • 24
  • 37
2

A simple workaround to the error is to use the class name as variable name so that pickle can find it:

import sys, pickle

class BC(object):
    pass

NewClassName = type("NewClassName", (BC,), {})

pickle.dump(NewClassName, sys.stdout)

However, this probably doesn't really do what you want. When loading the pickled class:

pickle.loads("""c__main__
NewClassName
p0
.""")

You again get the error:

AttributeError: 'module' object has no attribute 'NewClassName'

unless you've already defined the class.

As the documentation states:

pickle can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored.

So you can't use it to generate new classes, just to make sure your objects refer to the correct classes.

There are workarounds like pickling type parameters as shown in the other answer, but even then you will not be able to pickle objects of those dynamic classes without exposing the class in the global namespace of both the pickling and the unpickling process (i.e. __main__.ClassName must refer to the class).

Therefore, I would rethink the whole dynamic class approach.

Community
  • 1
  • 1
otus
  • 5,572
  • 1
  • 34
  • 48
  • There is now no variable `C` to pickle, should be: `pickle.dump(NewClassName, sys.stdout)` – Jamie Cockburn Jun 23 '14 at 11:20
  • @JamieCockburn, indeed, typo. Anyway, this "solves" the error, but probably not what the user wants to accomplish. – otus Jun 23 '14 at 11:22
  • Thx for your fast reply! The problem is, that C is a fix variable name and can't be changed depending on "NewClassName". – eatdas Jun 23 '14 at 11:26
  • 1
    @eatdas, see my update to the answer. If you are trying to load dynamically generated classes that don't exist in the "current" state, `pickle` alone won't work for you. – otus Jun 23 '14 at 11:28
  • 2
    @otus You are correct, it's telling that the pickled class definition makes no reference to the class's bases. The infomation to recreate the class definition is simply not saved. – Jamie Cockburn Jun 23 '14 at 11:30
2

You could use dill, which can serialize dynamic class definitions. Then you don't need any workarounds, and you can do exactly what you wanted to do.

Python 2.7.7 (default, Jun  2 2014, 01:33:50) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>>>
>>> class BC(object):
...   pass
... 
>>> c = type("NewClassName", (BC,), {})
>>> _c = dill.dumps(c)    
>>> c2 = dill.loads(_c)
>>> c2
<class '__main__.NewClassName'>
>>>

Get dill here: https://github.com/uqfoundation

Mike McKerns
  • 33,715
  • 8
  • 119
  • 139