11

Say I'm making a game with items in it (think about Minecraft, CS:GO weapons, LoL and Dota items, etc.). There can be huge amounts of the same item in the game with minor detail differences like the condition/durability or the amount of ammo remaining in the item:

player1.give_item(Sword(name='Sword', durability=50))
player2.give_item(Sword(name='Sword', durability=80))
player2.give_item(Pistol(name='Pistol', ammo=12))

But since I don't want to name my swords and pistols every time (due to the name always being the same), and I want it to be extremely easy for one to create new item classes, I figured I'd make name a class attribute:

class Item:
    name = 'unnamed item'

Now I simply subclass this:

class Sword(Item):
    name = 'Sword'

    def __init__(self, durability=100):
        self.durability = durability

class Pistol(Item):
    name = 'Pistol'

    def __init__(self, ammo=10):
        self.ammo = ammo

And we have working classes:

>>> sword = Sword(30)
>>> print(sword.name, sword.durability, sep=', ') 
Sword, 30

But is there a way to use these class attributes (and sometimes even classproperties) with SQLAlchemy in one way or another? Say, I want to store an item's durability (instance attribute) and name (class attribute) with its class_id (class property) as the primary key:

class Item:
    name = 'unnamed item'

    @ClassProperty  # see the classproperty link above
    def class_id(cls):
        return cls.__module__ + '.' + cls.__qualname__

class Sword(Item):
    name = 'Sword'

    def __init__(self, durability=100):
        self.durability = durability

The durability can easily be done with:

class Sword(Item):
    durability = Column(Integer)

But how about the name class attribute and class_id class property?

In reality I have much much larger inheritance tree and each class has multiple attributes/properties as well as more instance attributes.

UPDATE: I was unclear in my post about the tables. I only want to have one table for the items, where the class_id is used as the primary key. This is how I'd construct the table with metadata:

items = Table('items', metadata,
    Column('steamid', String(21), ForeignKey('players.steamid'), primary_key=True),
    Column('class_id', String(50), primary_key=True),
    Column('name', String(50)),
    Column('other_data', String(100)),  # This is __RARELY__ used for something like durability, so I don't need separate table for everything
)
Markus Meskanen
  • 19,939
  • 18
  • 80
  • 119
  • Are you asking if plain python attributes and declarative can be mixed, or have I misunderstood completely? – Ilja Everilä Jul 22 '16 at 12:03
  • @IljaEverilä I'm asking if there's a way to store class attributes and class properties to a database along the instance attributes. In my example code, I need to store item's ammo/durability bases on the item's `class_id`. Ammo and durability are instance attributes so I can just do `ammo = Column(Integer)`, but how would I do the class property (or even a class attribute)? – Markus Meskanen Jul 22 '16 at 12:43
  • I've updated the question to better explain the issue. – Markus Meskanen Jul 22 '16 at 12:51
  • When you say "along the instance attributes", do you mean "store them in each row (instance)" or something along the lines of having a separate table for "type" or class and then another for instances of said "type". `class_id` is a dubious primary key candidate, unless we're talking about said "type" table. – Ilja Everilä Jul 22 '16 at 21:07
  • If you're looking for having "class attributes" in a separate table, I'd say go with many to one relationships, possibly as [`declared_attr`](http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/mixins.html#mixing-in-relationships) from a mixin. Wrap it up with [association proxies](http://docs.sqlalchemy.org/en/latest/orm/extensions/associationproxy.html) to transparently access the attributes over the relationship, if need be. Still, there's nothing wrong in having the `name` attribute in your python class instead of a DB. – Ilja Everilä Jul 22 '16 at 21:33
  • @IljaEverilä Maybe this'll explain: I have an SQLite table with the following columns: `CREATE TABLE items (owner_id TEXT, class_id TEXT, durability INTEGER, PRIMARY KEY (steamid, class_id)` and these are received from the item like so: `(ìtem.owner.id, item.class_id, item.durability)` where `item.class_id` is the classproperty, and I don't know how to do the classproperty as a column in the Python class. – Markus Meskanen Jul 23 '16 at 09:06
  • I see. Make *class_id* a declared_attr returning a Column that defaults to what you're returning now. – Ilja Everilä Jul 23 '16 at 20:34
  • I'd update the answer to use class_id as polymorphic identity, `__mapper_args__` as declared_attr etc., but am behind a very limited connection for some time. – Ilja Everilä Jul 23 '16 at 20:45
  • @IljaEverilä I too was on holliday so didn't bother visiting SO that much. Thanks for all the help, I'll look into the terms you mentioned. If you're capable of updating your answer to use the custom attributes, I'd love to accept it too. – Markus Meskanen Jul 26 '16 at 09:08
  • Back from the wilderness. It seems @Julian has provided a great answer, so there's not much point in updating my own answer anymore. – Ilja Everilä Jul 31 '16 at 18:13
  • Welcome back, Ilja. For what it's worth, I liked your answer, too. :-) – Julian Aug 01 '16 at 12:00

3 Answers3

5

This is my second answer, based on single table inheritance.

The question contains an example where Item subclasses have their own specific instance attributes. For example, Pistol is the only class in the inheritance hierarchy that has an ammo attribute. When representing this in a database, you can save space by creating a table for the parent class that contains a column for each of the common attributes, and storing attributes that are specific to a subclass in a separate table for each of the subclasses. SQLAlchemy supports this out of the box and calls it joined table inheritance (because you need to join tables in order to collect both the common attributes and the attributes that are particular to a subclass). The answer by Ilja Everilä and my previous answer both assumed that joined table inheritance was the way to go.

As it turns out, Markus Meskanen's actual code is a bit different. The subclasses do not have particular instance attributes, they all just have a level attribute in common. Also, Markus commented that he wants all subclasses to be stored in the same table. A possible advantage of using a single table is that you can add and remove subclasses without causing major changes to the database schema every time.

SQLAlchemy offers support for this too, and it is called single table inheritance. It even works if the subclasses do have particular attributes. It is just a bit less efficient, because every row has to store every possible attribute even when it belongs to an item of a different subclass.

Here is a slightly altered version of solution 1 from my previous answer (originally copied from Ilja's answer). This version ("solution 1B") uses single table inheritance, so all items are stored in the same table.

class Item(Base):
    name = 'unnamed item'

    @classproperty
    def class_id(cls):
        return '.'.join((cls.__module__, cls.__qualname__))

    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    type = Column(String(50))
    durability = Column(Integer, default=100)
    ammo = Column(Integer, default=10)

    __mapper_args__ = {
        'polymorphic_identity': 'item',
        'polymorphic_on': type
    }


class Sword(Item):
    name = 'Sword'

    __mapper_args__ = {
        'polymorphic_identity': 'sword',
    }


class Pistol(Item):
    name = 'Pistol'

    __mapper_args__ = {
        'polymorphic_identity': 'pistol',
    }

When we compare this to the original solution 1, a few things stand out. The durability and ammo attributes have moved to the Item base class, so every instance of Item or one of its subclasses now has both a durability and an ammo. The Sword and Pistol subclasses have lost their __tablename__s as well as all of their column attributes. This is telling SQLAlchemy that Sword and Pistol do not have associated tables of their own; in other words, that we want to use single table inheritance. The Item.type column attribute and the __mapper_args__ business are still there; these provide the information for SQLAlchemy to determine whether any given row in the item table belongs to the Item, Sword or Pistol class. This is what I mean when I say that the type column is the disambiguator.

Now, Markus also commented he does not want to customize the subclasses in order to create a database mapping with single table inheritance. Markus wants to start with an existing class hierarchy with no database mapping and then create the entire single table inheritance database mapping at once by just editing the base class. That would mean that adding __mapper_args__ to the Sword and Pistol subclasses, like in solution 1B above, is out of the question. Indeed, if the disambiguator can be computed "automagically", this saves a lot of boilerplate, especially if there are many subclasses.

This can be done, using @declared_attr. Enter solution 4:

class Item(Base):
    name = 'unnamed item'

    @classproperty
    def class_id(cls):
        return '.'.join((cls.__module__, cls.__qualname__))

    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    type = Column(String(50))
    durability = Column(Integer, default=100)
    ammo = Column(Integer, default=10)

    @declared_attr
    def __mapper_args__(cls):
        if cls == Item:
            return {
                'polymorphic_identity': cls.__name__,
                'polymorphic_on': type,
            }
        else:
            return {
                'polymorphic_identity': cls.__name__,
            }


class Sword(Item):
    name = 'Sword'


class Pistol(Item):
    name = 'Pistol'

This yields the same result as solution 1B, except that the value of the disambiguator (still the type column) is computed from the class instead of being an arbitrarily chosen string. Here, it is simply the name of the class (cls.__name__). We could have chosen the fully qualified name instead (cls.class_id) or even the custom name attribute (cls.name), if you can guarantee that every subclass overrides name. It does not really matter what you take as the value of the disambiguator, as long as there is a one-to-one mapping between the value and the class.

Community
  • 1
  • 1
Julian
  • 4,176
  • 19
  • 40
  • This looks *really* nice, I'll have to test it as soon as I get home! One question though, how exactly do the classes know which data is theirs, since the `class_id` isn't stored along the data? Is the `'polymorphic_identity'` stored somehow? – Markus Meskanen Jul 30 '16 at 08:45
  • 1
    Yes, the `'polymorphic_identity'` is stored in the `type` column. That's what `'polymorphic_on': type` means. SQLAlchemy is using this behind your back to determine which class to instantiate. As I said, you could store the `class_id` in the `type` column, but it doesn't matter as long as you have a value that is uniquely tied to the class. – Julian Jul 30 '16 at 09:05
  • Awesome, thanks man! I'll test this later tonight or tomorrow, I'll report back as soon as I'm done testing :) – Markus Meskanen Jul 30 '16 at 09:51
  • I can't test on mobile, but as it stands I believe I still have to define the `type` column every time for the instances? `sword = Sword(type='Sword')`, right? How do I automate this process? – Markus Meskanen Jul 30 '16 at 12:14
  • 2
    No, you don't need to do anything except for the code inside the `Item` class. SQLAlchemy takes care of the `type` column for you, in fact you probably better never touch it. So you can just create a sword using `mysword = Sword()`. I'm glad it seems to be too good to be true, to you. ;-) – Julian Jul 30 '16 at 19:58
  • 1
    Oh wow that's awesome... Where does it get the value though? How does it know how to get the type? **Edit:** Oooh.. The `type` in `'polymorphic_identity': type` is the `type` function for getting an object's type, not the `type` variable in `type = Column()`... That's cool, thanks a lot for your answer! I'll accept tomorrow probably when I can reach a computer :) – Markus Meskanen Aug 01 '16 at 09:53
  • I have a base class `Entity` which then both `Item` and `Skill` subclass. This `Entity` class is an abstract base class, so I don't need an `entities` table. Can I move the database related code into the `Entity` class and somehow skip the `__tablename__` which would then be defined separately in `Item` and `Skill` subclasses (which are still baseclasses of many skills and items)? Or should I just do `class Skill(Entity, Base):` for both `Skill` and `Item` and ignore the few lines of duplicate code? – Markus Meskanen Aug 01 '16 at 10:09
  • Here's what I mean... [This code](http://pastebin.com/E2ZHk9GD) works but has duplicate code in `Skill` and `Item`. Whereas [this](http://pastebin.com/GaNUJhbF) code with no duplicate code results into `sqlalchemy.exc.NoForeignKeysError: Can't find any foreign key relationships between 'entities' and 'skills'.`. But I don't want there to be a foreign key, I don't even need the `entities` table at all. – Markus Meskanen Aug 01 '16 at 11:06
  • The problem in your second paste is that `Entity` derives from `Base`. This makes SQLA believe that you want to map it to a table. Both `Entity` and `Skill` declare columns, so it assumes joined table inheritance; hence the "no foreign keys" complaint. Fortunately, you can just *not* derive `Entity` from `Base` and use it as a [mixin](http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/mixins.html). Something like [this](http://pastebin.com/vNCPr4Ni) should work. The `_declarative_constructor` is there to save you the effort of writing an `__init__` for every subclass of `Entity`. – Julian Aug 01 '16 at 11:51
  • I'd give you a long thank you speech, but I suppose you already know how thankful I am for the huge amount of help. Seriously, thanks, I couldn't have learned all this without you. – Markus Meskanen Aug 01 '16 at 16:02
  • Welcome. Glad I could help you. :-) – Julian Aug 01 '16 at 20:28
2

Quoting the official documentation:

When our class is constructed, Declarative replaces all the Column objects with special Python accessors known as descriptors; ...

Outside of what the mapping process does to our class, the class remains otherwise mostly a normal Python class, to which we can define any number of ordinary attributes and methods needed by our application.

From that it should be clear that adding class attributes, methods etc. is possible. There are certain reserved names though, namely __tablename__, __table__, metadata and __mapper_args__ (not an exhaustive list).

As for inheritance, SQLAlchemy offers three forms: single table, concrete and joined table inheritance.

Implementing your simplified example using joined table inheritance:

class Item(Base):
    name = 'unnamed item'

    @classproperty
    def class_id(cls):
        return '.'.join((cls.__module__, cls.__qualname__))

    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    type = Column(String(50))

    __mapper_args__ = {
        'polymorphic_identity': 'item',
        'polymorphic_on': type
    }


class Sword(Item):
    name = 'Sword'

    __tablename__ = 'sword'
    id = Column(Integer, ForeignKey('item.id'), primary_key=True)
    durability = Column(Integer, default=100)

    __mapper_args__ = {
        'polymorphic_identity': 'sword',
    }


class Pistol(Item):
    name = 'Pistol'

    __tablename__ = 'pistol'
    id = Column(Integer, ForeignKey('item.id'), primary_key=True)
    ammo = Column(Integer, default=10)

    __mapper_args__ = {
        'polymorphic_identity': 'pistol',
    }

Adding items and querying:

In [11]: session.add(Pistol())

In [12]: session.add(Pistol())

In [13]: session.add(Sword())

In [14]: session.add(Sword())

In [15]: session.add(Sword(durability=50))

In [16]: session.commit()

In [17]: session.query(Item).all()
Out[17]: 
[<__main__.Pistol at 0x7fce3fd706d8>,
 <__main__.Pistol at 0x7fce3fd70748>,
 <__main__.Sword at 0x7fce3fd709b0>,
 <__main__.Sword at 0x7fce3fd70a20>,
 <__main__.Sword at 0x7fce3fd70a90>]

In [18]: _[-1].durability
Out[18]: 50

In [19]: item =session.query(Item).first()

In [20]: item.name
Out[20]: 'Pistol'

In [21]: item.class_id
Out[21]: '__main__.Pistol'
Community
  • 1
  • 1
Ilja Everilä
  • 50,538
  • 7
  • 126
  • 127
1

The answer by Ilja Everilä is already a best possible. While it does not store the value of class_id inside the table literally, please observe that any two instances of the same class always have the same value of class_id. So knowing the class is sufficient to compute the class_id for any given item. In the code example that Ilja provided, the type column ensures that the class can always be known and the class_id class property takes care of the rest. So the class_id is still represented in the table, if indirectly.

I repeat Ilja's example from his original answer here, in case he decides to change it in his own post. Let us call this "solution 1".

class Item(Base):
    name = 'unnamed item'

    @classproperty
    def class_id(cls):
        return '.'.join((cls.__module__, cls.__qualname__))

    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    type = Column(String(50))

    __mapper_args__ = {
        'polymorphic_identity': 'item',
        'polymorphic_on': type
    }


class Sword(Item):
    name = 'Sword'

    __tablename__ = 'sword'
    id = Column(Integer, ForeignKey('item.id'), primary_key=True)
    durability = Column(Integer, default=100)

    __mapper_args__ = {
        'polymorphic_identity': 'sword',
    }


class Pistol(Item):
    name = 'Pistol'

    __tablename__ = 'pistol'
    id = Column(Integer, ForeignKey('item.id'), primary_key=True)
    ammo = Column(Integer, default=10)

    __mapper_args__ = {
        'polymorphic_identity': 'pistol',
    }

Ilja hinted at a solution in his last comment to the question, using @declared_attr, which would literally store the class_id inside the table, but I think it would be less elegant. All it buys you is representing the exact same information in a slightly different way, at the cost of making your code more complicated. See for yourself ("solution 2"):

class Item(Base):
    name = 'unnamed item'

    @classproperty
    def class_id_(cls):  # note the trailing underscore!
        return '.'.join((cls.__module__, cls.__qualname__))

    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    class_id = Column(String(50))  # note: NO trailing underscore!

    @declared_attr  # the trick
    def __mapper_args__(cls):
        return {
            'polymorphic_identity': cls.class_id_,
            'polymorphic_on': class_id
        }


class Sword(Item):
    name = 'Sword'

    __tablename__ = 'sword'
    id = Column(Integer, ForeignKey('item.id'), primary_key=True)
    durability = Column(Integer, default=100)

    @declared_attr
    def __mapper_args__(cls):
        return {
            'polymorphic_identity': cls.class_id_,
        }


class Pistol(Item):
    name = 'Pistol'

    __tablename__ = 'pistol'
    id = Column(Integer, ForeignKey('item.id'), primary_key=True)
    ammo = Column(Integer, default=10)

    @declared_attr
    def __mapper_args__(cls):
        return {
            'polymorphic_identity': cls.class_id_,
        }

There is also an additional danger in this approach, which I will discuss later.

In my opinion, it would be more elegant to make the code simpler. This could be achieved by starting with solution 1 and then merging the name and type properties, since they are redundant ("solution 3"):

class Item(Base):
    @classproperty
    def class_id(cls):
        return '.'.join((cls.__module__, cls.__qualname__))

    __tablename__ = 'item'
    id = Column(Integer, primary_key=True)
    name = Column(String(50))  # formerly known as type

    __mapper_args__ = {
        'polymorphic_identity': 'unnamed item',
        'polymorphic_on': name,
    }


class Sword(Item):
    __tablename__ = 'sword'
    id = Column(Integer, ForeignKey('item.id'), primary_key=True)
    durability = Column(Integer, default=100)

    __mapper_args__ = {
        'polymorphic_identity': 'Sword',
    }


class Pistol(Item):
    __tablename__ = 'pistol'
    id = Column(Integer, ForeignKey('item.id'), primary_key=True)
    ammo = Column(Integer, default=10)

    __mapper_args__ = {
        'polymorphic_identity': 'Pistol',
    }

All three solutions discussed so far give you the exact same requested behaviour on the Python side (assuming that you will ignore the type attribute). For example, an instance of Pistol will return 'yourmodule.Pistol' as its class_id and 'Pistol' as its name in each solution. Also in each solution, if you add a new item class to the hierarchy, say Key, all its instances will automatically report their class_id to be 'yourmodule.Key' and you will be able to set their common name once at the class level.

There are some subtle differences on the SQL side, regarding the name and value of the column that disambiguates between the item classes. In solution 1, the column is called type and its value is arbitrarily chosen for each class. In solution 2, the column name is class_id and its value is equal to the class property, which depends on the class name. In solution 3, the name is name and its value is equal to the name property of the class, which can be varied independently from the class name. However, since all these different ways to disambiguate the item class can be mapped one-to-one to each other, they contain the same information.

I mentioned before that there is a catch in the way solution 2 disambiguates the item class. Suppose that you decide to rename the Pistol class to Gun. Gun.class_id_ (with trailing underscore) and Gun.__mapper_args__['polymorphic_identity'] will automatically change to 'yourmodule.Gun'. However, the class_id column in your database (mapped to Gun.class_id without trailing underscore) will still contain 'yourmodule.Pistol'. Your database migration tool might not be smart enough to figure out that those values need to be updated. If you are not careful, your class_ids will be corrupted and SQLAlchemy will likely throw exceptions at you for being unable to find matching classes for your items.

You could avoid this problem by using an arbitrary value as the disambiguator, as in solution 1, and storing the class_id in a separate column using @declared_attr magic (or a similar indirect route), as in solution 2. However, at this point you really need to ask yourself why the class_id needs to be in the database table. Does it really justify making your code so complicated?

Take home message: you can map plain class attributes as well computed class properties using SQLAlchemy, even in the face of inheritance, as illustrated by the solutions. That does not necessarily mean you should actually do it. Start with your end goals in mind, and find the simplest way to achieve those goals. Only make your solution more sophisticated if doing so solves a real problem.

Community
  • 1
  • 1
Julian
  • 4,176
  • 19
  • 40
  • Hey! Before answering to the rest of your answer, I'd like to talk about storing the `class_id`. I might've been unclear in my original question, but I *don't* want a separate table for each subclass, I just want one `items` table and thus I believe the `class_id` is necessary to identify which item is for which class. So I'm looking to get rid of *all* the detabase functionality from the subclasses, and have it all done in the `Item` base class. I hope that makes sense! – Markus Meskanen Jul 29 '16 at 06:58
  • [Here](https://github.com/Mahi/RPG-SP/blob/master/addons/source-python/plugins/rpg/database.py) is my existing database functionality using nothing but `sqlite3` and direct `SQL` commands, but I'd like to switch over to SQLAlchemy. [These lines](https://github.com/Mahi/RPG-SP/blob/master/addons/source-python/plugins/rpg/rpg.py#L25-L89) pretty much show how I'm currently using the custom database class. And finally, [here's what my skills (~items) look like](https://github.com/Mahi/RPG-SP/blob/master/addons/source-python/plugins/rpg/skills.py), and this is what I'd like to keep as-is. – Markus Meskanen Jul 29 '16 at 07:00
  • You want a single table to store all items of the different subclasses. That calls for single table inheritance in SQLAlchemy parlance (Ilja already mentioned the different types of inheritance support in SQLA). So far, we have been discussing joined table inheritance, but the code for single table inheritance looks very similar. In single table inheritance, the disambiguator still doesn't have to be the `class_id`. Would you like me to write a new answer, demonstrating the code for single table inheritance? – Julian Jul 29 '16 at 09:25
  • I don't see how it can be anything other than `class_id`, but I'm no expert with SQLAlchemy so not going to argue. I tried the current answers and they don't work the way I want them to, as they force me to customize my subclasses (the skills.py file I linked to in my previous comment). I would love a separate answer if you don't mind! I'm actively working on this project and I've been stuck here for days now, trying to gasp through the SQLAlchemy docs. I'd increase the bounty if I could... – Markus Meskanen Jul 29 '16 at 09:30
  • You can start a second bounty when the current one is over. I wouldn't mind. ;-) I'll write a second answer demonstrating single table inheritance. I'll discuss the options, including one that allows you to leave the `Skill` subclasses alone. I'll try to do it today, but otherwise it will be tomorrow. – Julian Jul 29 '16 at 10:29
  • If your second solution is as awesome as I hope, I just might increase it! And sure, no rush obviously :) – Markus Meskanen Jul 29 '16 at 13:06