Dynamic Python Class Definition in SQLAlchemy

Question

I'm creating a backend application with SQLAlchemy using the declarative base. The ORM requires about 15 tables each of which maps to a class object in SQLAlchemy. Because these class objects are all defined identically I thought a factory pattern could produce the classes more concisely. However, these classes not only have to be defined, they have to be assigned to unique variable names so they can be imported and used through the project.

(Sorry if this question is a bit long, I updated it as I better understood the problem.)

Because we have so many columns (~1000) we define their names and types in external text files to keep things readable. Having done that one way to go about declaring our models is like this:

class Foo1(Base):
    __tablename___ = 'foo1'

class Foo2(Base):
    __tablename___ = 'foo2'

... etc

and then I can add the columns by looping over the contents of the external text file and using the setattr() on each class definition.

This is OK but it feels too repetitive as we have about 15 tables. So instead I took a stab at writing a factory function that could define the classes dynamically.

def orm_factory(class_name):
    class NewClass(Base):
        __tablename__ = class_name.lower()
    NewClass.__name__ = class_name.upper()
    return NewClass

Again I can just loop over the columns and use setattr(). When I put it together it looks like this:

for class_name in class_name_list:
    ORMClass = orm_factory(class_name)
    header_keyword_list = get_header_keyword_list(class_name)
    define_columns(ORMClass, header_keyword_list)

Where get_header_keyword_list gets the column information and define_columns performs the setattr() assignment. When I use this and run Base.metadata.create_all() the SQL schema get generated just fine.

But, when I then try to import these class definitions into another model I get an error like this:

SAWarning: The classname 'NewClass' is already in the registry of this declarative base, mapped to <class 'ql_database_interface.IR_FLT_0'>

This, I now realize makes total sense based on what I learned yesterday: Python class variable name vs __name__.

You can address this by using type as a class generator in your factory function (as two of the answers below do). However, this does not solve the issue of being able to import the class because the while the classes are dynamically constructed in the factory function the variable the output of that function is assigned to is static. Even if it were dynamic, such as a dictionary key, it has to be in the module name space in order to be imported from another module. See my answer for more details.

score 2 · Answer 1 · answered Aug 22 '13 at 01:47

This sounds like a sketchy idea. But it's fun to solve so here is how you make it work.

As I understand it, your problem is you want to add dynamically created classes to a module. I created a hack using a module and the init.py file.

dynamicModule/__init__.py:

import dynamic

class_names = ["One", "Two", "Three"]

for new_name in class_names:
     dynamic.__dict__['Class%s' % new_name] = type("Class%s" % (new_name), (object,), {'attribute_one': 'blah'})

dynamicModule/dynamic.py:

"""Empty file"""

test.py:

import dynamicModule
from dynamicModule import dynamic
from dynamicModule.dynamic import ClassOne

dynamic.ClassOne
"""This all seems evil but it works for me on python 2.6.5"""

__init__.py:

"""Empty file"""

score 1 · Accepted Answer · edited May 23 '17 at 10:28

[Note, this is the original poster]

So after some thinking and talking to people I've decided that that ability to dynamically create and assign variables to class objects in the global name space in this way this just isn't something Python supports (and likely with good reason). Even though I think my use case isn't too crazy (pumping out predefined list of identically constructed classes) it's just not supported.

There are lots of questions that point towards using a dictionary in a case like this, such as this one: https://stackoverflow.com/a/10963883/1216837. I thought of something like that but the issue is that I need those classes in the module name space so I can import them into other modules. However, adding them with globals() like globals()['MyClass'] = class_dict['MyClass'] seems like it's getting pretty out there and my impression is people on SO frown on using globals() like this.

There are hacks such as the one suggested by patjenk but at a certain point the obfuscation and complexity out weight the benefits of the clarity of declaring each class object statically. So while it seems repetitive I'm just going to write out all the class definitions. Really, this end up being pretty concise/maintainable:

Class1 = class_factory('class1')
Class2 = class_factory('class2')
...

Why do you need to modify globals if you add the constructed classes to a outer-scope container (list_of_classes.append or a dict)? The container itself will be importable across modules because the container will be in the module scope. — cowbert, Oct 13 '17 at 19:06

Dynamic Python Class Definition in SQLAlchemy

2 Answers2

Linked