How can I serialize a MongoDB ObjectId with Marshmallow?

Question

I'm building and API on top of Flask using marshmallow and mongoengine. When I make a call and an ID is supposed to be serialized I receive the following error:

TypeError: ObjectId('54c117322053049ba3ef31f3') is not JSON serializable

I saw some ways with other libraries to override the way the ObjectId is treated. I haven't figured it out with Marshmallow yet, does anyone know how to do that?

My model is:

class Process(db.Document):
    name = db.StringField(max_length=255, required=True, unique=True)
    created_at = db.DateTimeField(default=datetime.datetime.now, required=True)

My serializer:

class ProcessSerializer(Serializer):
    class Meta:
        fields = ("id", "created_at", "name")

And the view:

class ProcessView(Resource):
    def get(self, id):
        process = Process.objects.get_or_404(id)
        return ProcessSerializer(process).data

Answer 1

When you just pass Meta.fields to a schema, Marshmallow tries to pick a field type for each attribute. Since it doesn't know what an ObjectId is, it just passes it on to the serialized dict. When you try to dump this to JSON, it doesn't know what an ObjectId is and raises an error. To solve this, you need to tell Marshmallow what field to use for the id. A BSON ObjectId can be converted to a string, so use a String field.

from marshmallow import Schema, fields

class ProcessSchema(Schema):
    id = fields.String()

    class Meta:
        additional =  ('created_at', 'name')

You can also tell Marshmallow what field to use for the ObjectId type so that you don't have to add the field each time.

from bson import ObjectId
from marshmallow import Schema, fields

Schema.TYPE_MAPPING[ObjectId] = fields.String

Answer 2

marshmallow-mongoengine does this:

Marshmallow-Mongoengine is about bringing together a Mongoengine Document with a Marshmallow Schema.

import marshmallow_mongoengine as ma


class ProcessSchema(ma.ModelSchema):
    class Meta:
        model = Process

It has an ObjectId field that serializes/deserializes ObjectIds.

Answer 3

You can extend the fields.Field class to create your own field. Here's how marshmallow-mongoengine (mentioned in another answer) implements this:

import bson
from marshmallow import ValidationError, fields, missing

class ObjectId(fields.Field):
    def _deserialize(self, value, attr, data):
        try:
            return bson.ObjectId(value)
        except Exception:
            raise ValidationError("invalid ObjectId `%s`" % value)

    def _serialize(self, value, attr, obj):
        if value is None:
            return missing
        return str(value)

and then:

class MySchema(Schema):
    id = ObjectId()

(I found this useful when not using MongoEngine, just using pymongo)

Answer 4

Similar to @dcollien above, I extended fields.Field and created my own custom field with helpers, similar to how Marshmallow handles field types internally:

from marshmallow import fields, missing
from marshmallow.exceptions import ValidationError
from bson.objectid import ObjectId
from bson.errors import InvalidId
import json

def oid_isval(val: Any) -> bool:
    """
    oid_isval [summary]

    Parameters
    ----------
    val : {Any}
        Value to be assessed if its an ObjectId

    Returns
    ----------
    val : bool
        True if val is an ObjectId, otherwise false
    """    
    if ObjectId.is_valid(val):
        return val

def ensure_objid_type(val: Union[bytes, str, ObjectId]) -> ObjectId:
    """
    Ensures that the value being passed is return as an ObjectId and is a valid ObjectId

    Parameters
    ----------
    val : Union[bytes, str, ObjectId]
        The value to be ensured or converted into an ObjectId and is a valid ObjectId

    Returns
    ----------
    val : ObjectId
        Value of type ObjectId
        
    Raises
    ----------
    ValidationError: Exception
        If it's not an ObjectId or can't be converted into an ObjectId, raise an error.
            
    """
    try:
        # If it's already an ObjectId and it's a valid ObjectId, return it
        if isinstance(val, ObjectId) and oid_isval(val):
            logger.info(f"It's an ObjectId and it's valid! = {val}")
            return val
        
        # Otherwise, if it's a bytes object, decode it and turn it into a string
        elif isinstance(val, bytes):
            val = ObjectId(str(val.decode("utf-8")))
            logger.info(f"Decoded and converted bytes object to ObjectId! = {val}")

        # Otherwise, if it's a string, turn it into an ObjectId and check that it's valid 
        elif isinstance(val, str):
            val = ObjectId(val)
            logger.info(f"Converted str to ObjectId! = {val}")
        
        # Check to see if the converted value is a valid objectId
        if oid_isval(val):
            logger.info(f"It's a valid ObjectId! = {val}")
            return val
    except InvalidId as error:
        logger.error(f"Not a valid ObjectId = {val} | error = {error}")
        raise ValidationError(json.loads(json.dumps(f"{error}")))

class ObjectIdField(fields.Field):
    """Custom field for ObjectIds."""
    # Default error messages
    default_error_messages = {
        "invalid_ObjectId": "Not a valid ObjectId."
    }

    def _serialize(self, value, attr, obj, **kwargs) -> Optional[ObjectId]:
        if value is None:
            return None
        return ensure_objid_type(value)

    def _deserialize(self, value, attr, data, **kwargs):
        if value is None:
            return missing
        if not isinstance(value, (ObjectId, str, bytes)):
            raise self.make_error("_deserialize: Not a invalid ObjectId")
        try:
            return ensure_objid_type(value)
        except UnicodeDecodeError as error:
            raise self.make_error("invalid_utf8") from error
        except (ValueError, AttributeError, TypeError) as error:
            raise ValidationError("ObjectIds must be a 12-byte input or a 24-character hex string") from error