How to dynamically generate marshmallow schemas for SQLAlchemy models

Question

I'm creating a Flask API using SQLAlchemy models. I don't want to define a schema for every model I have, I don't want to do this every time:

class EntrySchema(ma.ModelSchema):
    class Meta:
        model = Entry

I would like each model to have a schema, so it can easily dump itself. Creating a default Schema and setting the Schema.Meta.model didn't work:

class Entry(db.Model):
    __tablename__ = 'entries'

    id = db.Column(db.Integer, primary_key=True)
    started_at = db.Column(db.DateTime)
    ended_at = db.Column(db.DateTime)
    description = db.Column(db.Text())

    def __init__(self, data):
        for key in data:
            setattr(self, key, data[key])

        self.Schema = Schema
        self.Schema.Meta.model = self.__class__

    def dump(self):
        schema = self.Schema()
        result = schema.dump(self)
        return result


class Schema(ma.ModelSchema):
    class Meta:
        pass

Why is a generic Schema with the model overwritten different than a Schema with the model declared?

Ilja Everilä · Accepted Answer · 2020-09-02T05:58:19.447

15

You could create a class decorator that adds the Schema to your models:

def add_schema(cls):
    class Schema(ma.ModelSchema):
        class Meta:
            model = cls
    cls.Schema = Schema
    return cls

and then

@add_schema
class Entry(db.Model):
    ...

The schema will be available as the class attribute Entry.Schema.

The reason your original attempt fails is that marshmallow Schema classes are constructed using a custom metaclass, which inspects the namespace created from executing the class body and does its thing. When you modify the already constructed class, it is too late.

If you're unfamiliar with metaclasses in Python, read about them in the language reference. They are a tool that allows for great things and great misuse.

Some more complex types, such as enums, require additional information and dedicated field types to work properly. For example using marshmallow-enum and a decorator factory pattern it is possible to configure the model schema to accommodate enums:

from marshmallow_enum import EnumField

def add_schema(**kwgs):
    def decorator(cls): 
        class Meta:
            model = cls

        schema = type("Schema", (ma.ModelSchema,), {"Meta": Meta, **kwgs})
        cls.Schema = schema
        return cls

    return decorator

...


@add_schema(
    my_enum=EnumField(MyEnumType, by_value=True)
)
class Entry(db.Model):
    ...

Of course another way would be to make the decorator itself smarter and inspect the class before building the schema, so that it handles special cases such as enums.

edited Sep 02 '20 at 05:58

answered Mar 19 '17 at 21:22

Ilja Everilä

50,538
7
126
127

Can you do it without sqlalchemy? I have some denormalized classes with lists in them but sqlalchemy does not allow lists. – Oleg Yablokov Jul 27 '20 at 10:26
1

UPD: found [this project](https://github.com/lovasoa/marshmallow_dataclass) that uses Python's dataclasses instead of sqlalchemy. And even more syntax sugar! – Oleg Yablokov Jul 27 '20 at 10:56
1

if the model has an Enum, an error will be thrown: `Object of type MyEnum is not JSON serializable` upon dumping – jgozal Sep 02 '20 at 01:51
@jgozal That is unfortunate, but not strictly related to the decoration itself, but to how marshmallow works – and I am not that familiar with it in the end. If it requires extra information to make it correctly serialize enums, you would then turn `add_schema` to a decorator factory that accepts the necessary arguments, and returns the "configured" decorator closure. – Ilja Everilä Sep 02 '20 at 05:15
@jgozal Added an example of the above. – Ilja Everilä Sep 02 '20 at 05:28
1

thanks @IljaEverilä. This is a solid solution. I'm working on a way to find the enums in the schema by looking at the _declared_fields after the schema is created and then turning each of them into marshmallow enum fields dynamically. I'll let you know if I can make it work! I posted a question here as well: https://stackoverflow.com/questions/63697476/generating-marshmallow-schema-automatically-with-json-serializable-enums – jgozal Sep 02 '20 at 05:32
@jgozal If you are taking the inspection route, I highly recommend using the [runtime inspection API](https://docs.sqlalchemy.org/en/13/core/inspection.html) and tools, instead of accessing the protected attributes directly (though, is `_declared_fields` a marshmallow thing?). – Ilja Everilä Sep 02 '20 at 05:55
@IljaEverilä I believe `_declared_fields` is a marshmallow thing, but it's still a protected attribute? I've come up with a more dynamic solution, please see my answer in the question I referenced in my earlier comment. I'd love to get your thoughts, and you can hopefully help me improve it as well. – jgozal Sep 02 '20 at 06:29
Usually the `_` is used to indicate that an attribute is an implementation detail, and should not be relied on, but sometimes you cannot help it. – Ilja Everilä Sep 02 '20 at 06:31

Avishay116 · Answer 2 · 2019-05-20T17:24:02.687

From marshmallow-sqlalchemy recipes:

"Automatically Generating Schemas For SQLAlchemy Models It can be tedious to implement a large number of schemas if not overriding any of the generated fields as detailed above. SQLAlchemy has a hook that can be used to trigger the creation of the schemas, assigning them to the SQLAlchemy model property ".

My example using flask_sqlalchemy & marshmallow_sqlalchemy:

from flask_sqlalchemy import SQLAlchemy
from marshmallow_sqlalchemy import ModelConversionError, ModelSchema
from sqlalchemy import event
from sqlalchemy.orm import mapper


db = SQLAlchemy()


def setup_schema(Base, session):
    # Create a function which incorporates the Base and session information
    def setup_schema_fn():
        for class_ in Base._decl_class_registry.values():
            if hasattr(class_, "__tablename__"):
                if class_.__name__.endswith("Schema"):
                    raise ModelConversionError(
                        "For safety, setup_schema can not be used when a"
                        "Model class ends with 'Schema'"
                    )

                class Meta(object):
                    model = class_
                    sqla_session = session

                schema_class_name = "%sSchema" % class_.__name__

                schema_class = type(schema_class_name, (ModelSchema,), {"Meta": Meta})

                setattr(class_, "Schema", schema_class)

    return setup_schema_fn


event.listen(mapper, "after_configured", setup_schema(db.Model, db.session))

There is another example in the recipes:

https://marshmallow-sqlalchemy.readthedocs.io/en/latest/recipes.html#automatically-generating-schemas-for-sqlalchemy-models

A really nice approach, and a good example of SQLA events in general. — Ilja Everilä, Sep 02 '20 at 06:00

score 1 · Answer 3 · answered Jan 09 '18 at 19:42

The marshmallow recipes prescribe a couple of alternative options for throwing common schema options into a base class. Here's a quick example straight from the docs:

# myproject/schemas.py

from marshmallow_sqlalchemy import ModelSchema

from .db import Session

class BaseSchema(ModelSchema):
    class Meta:
        sqla_session = Session

and then extend the base schema:

# myproject/users/schemas.py

from ..schemas import BaseSchema
from .models import User

class UserSchema(BaseSchema):

    # Inherit BaseSchema's options
    class Meta(BaseSchema.Meta):
        model = User

The advantage of this approach is that you can add more de/serialization to specific models

More examples and recipes on the linked docs

How to dynamically generate marshmallow schemas for SQLAlchemy models

3 Answers3

Linked