Add database support at runtime

Question

I have a python module that I've been using over the years to process a series of text files for work. I now have a need to store some of the info in a db (using SQLAlchemy), but I would still like the flexibility of using the module without db support, i.e. not have to actually have sqlalchemy import'ed (or installed). As of right now, I have the following... and I've been creating Product or DBProduct, etc depending on wether I intend to use a db or not.

from sqlalchemy.ext.declarative import declarative_base

Base = declarative_base()

class Product(object):
    pass

class WebSession(Product):
    pass

class Malware(WebSession):
    pass

class DBProduct(Product, Base):
    pass

class DBWebSession(WebSession, DBProduct):
    pass

class DBMalware(Malware, DBWebSession):
    pass

However, I feel that there has got to be an easier/cleaner way to do this. I feel that I'm creating an inheritance mess and potential problems down the road. Ideally, I'd like to create a single class of Product, WebSession, etc (maybe using decorators) that contains the information neccessary for using a db, but it's only enabled/functional after calling something like enable_db_support(). Once that function is called, then regardless of what object I create, itself (and all the objects it inherits) enable all the column bindings, etc. I should also note that if I somehow figure out how to include Product and DBProduct in one class, I sometimes need 2 versions of the same function: 1 which is called if db support is enabled and 1 if it's not. I've also considered "recreating" the object hierarchy when enable_db_support() is called, however that turned out to be a nightmare as well.

Any help is appreciated.

score 0 · Answer 1 · edited May 23 '17 at 12:04

It seems to me that the DRYest thing to do would be to abstract away the details of your data storage format, be that a plain text file or a database.

That is, write some kind of abstraction layer that your other code uses to store the data, and make it so that the output of your abstraction layer is switchable between SQL or text.

Or put yet another way, don't write a Product and DB_Product class. Instead, write a store_data() function that can be told to use either format='text' or format='db'. Then use that everywhere.

This is actually the same thing SQLAlchemy does behind the scenes - you don't have to write separate code for SQLAlchemy depending on whether it's driving mySQL, PostgreSQL, etc. That is all handled in SQLAlchemy, and you use the abstracted (database-neutral) interface.

Alternately, if your objection to SQLAlchemy is that it's not a Python builtin, there's always sqlite3. This gives you all the goodness of an SQL relational database with none of the fat.

Alternately alternately, use sqlite3 as an intermediate format. So rewrite all your code to use sqlite3, and then translate from sqlite3 to plain text (or another database) as required. In the limit case, conversion to plain text is only a sqlite3 db .dump away.

The `store_data()` approach is a possibility, however I would lose the "seamless-ness" that `SQLAlchemy` provides. Additionally, I would have to remember to call the function and possibly have another one for updating.I don't have an objection to using SQLAlchemy and, in fact, that is what I'm using when it comes to making the data persistent in a database. However, there are many situations that I (or a coworker) don't need the data to be saved in a database. In those situations, I'd prefer that SQLAlchemy (or a db driver) not be a required install/import. — redlamb, Mar 19 '12 at 01:36

score 0 · Accepted Answer · answered Mar 17 '12 at 17:23

0

Well, you can probably get away with creating a pure non-DB aware model by using Classical Mapping without using a declarative extension. In this case, however, you will not be able to use relationships as they are used in SA, but for simple data import/export types of models this should suffice:

# models.py
class User(object):
    pass

----

# mappings.py
from sqlalchemy import Table, MetaData, Column, ForeignKey, Integer, String
from sqlalchemy.orm import mapper
from models import User

metadata = MetaData()

user = Table('user', metadata,
            Column('id', Integer, primary_key=True),
            Column('name', String(50)),
            Column('fullname', String(50)),
            Column('password', String(12))
        )

mapper(User, user)

Another option would be to have a base class for your models defined in some other module and configure on start-up this base class to be either DB-aware or not, and in case of DB-aware version add additional features like relationships and engine configurations...

answered Mar 17 '12 at 17:23

van

74,297
13
168
171

thanks for the response. This is a possibility that might work for me. I'll just have to look into the relationship issue. I tend to rely on `relationships` a lot, but it still might work. Thanks. – redlamb Mar 19 '12 at 01:39
The more I think of it, I think I considered going this approach, but didn't because of the need to potentially override some of a classes' defined functions when storing information in a database. Do you know if it's possible (or how I would go about doing it) to redefine some of a class' functions using this approach? Thanks again. – redlamb Mar 19 '12 at 02:09
@redlamb: well, due to the power of the mighty python, you can always add relationships in your `enable_db_support()` by just assigning method to a class: `DBProduct.versions = relationship('ProductVersion', ...)`. In the same way you can override methods: you just define a new (unbound) method like `def db_override_me(self, param1, ...): ...` and then (re-)assign it to you class: `DBProduct.override_me = db_override_me`. Although I fear there might be a better solution to this type of design... – van Mar 19 '12 at 07:13
I did take your approach, however after doing a bit more research, I was able to create relationships with [Classical Mappings](http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html#classical-mappings). You just need to supply the `property` argument when you [Configure the Mapper](http://docs.sqlalchemy.org/en/latest/orm/mapper_config.html). – redlamb Mar 31 '12 at 15:04

Add database support at runtime

2 Answers2