How can I manage changing data structures in MongoDb collections w/ Simple.Data?

Question

We're currently using Simple.Data and the MongoDb adapter. When we've retrieved a document, we cast it into a POCO, e.g:

(User)db.Users.FindById(1234);

To begin with, this works quite well (heck yeah, no schema!). However, if we change the structure of the User object (e.g. add a new field, or change a field's data type), then we can no longer cast the original document, as it doesn't match our new class structure.

To resolve this, we've so far tried the two most straight-forward approaches:

Manual data updates to reflect changes in document structure. Ok at the moment, but not manageable when the project is deployed across multiple environments / makes it into production
Manual mapping; eg. casting the SimpleRecord to a dictionary and evaluating members manually. I'm concerned about performance of this approach, though haven't bench-marked it yet. I'm also concerned that I haven't found a way to make it generic without using reflection on the destination type to identify member names.

We've also looked into ways this was solved with Ruby and Python. The former appeals more (the maintenance of old schema versions in Ming seems like it may be overkill).

Before I run off and port something crazy, has anyone solved this problem with Simple.Data? Can anyone offer any guidance as to best practices for dealing with document structure changes in schema-less databases?

It very much depends on conrete scenario, there is no "This is how it is done" way in mongo, so some use cases would be useful — Yurii Hohan, Sep 17 '11 at 19:28
@Hohhi Mostly have been adding fields or changing data types, eg. add a new field to a document `public List Activities`, or change type, `public int SomeValue` becomes `public string SomeValue`. There's only two devs at the moment, so we've been ok at managing this, but with more developers and more build, it seems like it will cause headaches. As far as I can see, it's a case of either changing the old data, or mapping in such a way as to ignore data changes, but I'm not really clear on how to make this relatively seamless / reduce pain in an environment with rapid changes... — Morgan Bruce, Sep 18 '11 at 01:16

score 2 · Accepted Answer · answered Sep 19 '11 at 20:01

Look through custom serialization. It was fast and useful in my case:

public class FieldsWrapper : IBsonSerializable
{
    public List<DataFieldValue> DataFieldValues { get; set; }


    public object Deserialize(MongoDB.Bson.IO.BsonReader bsonReader, Type nominalType, IBsonSerializationOptions options)
    {
    if (nominalType != typeof(FieldsWrapper)) throw new ArgumentException("Cannot deserialize anything but self");
    var doc = BsonDocument.ReadFrom(bsonReader);
    var list = new List<DataFieldValue>();
    foreach (var name in doc.Names)
    {
        var val = doc[name];
        if (val.IsString)
            list.Add(new DataFieldValue {LocalIdentifier = name, Values = new List<string> {val.AsString}});
        else if (val.IsBsonArray)
        {
            DataFieldValue df = new DataFieldValue {LocalIdentifier = name};
            foreach (var elem in val.AsBsonArray)
            {
                df.Values.Add(elem.AsString);
            }
            list.Add(df);
        }
    }
    return new FieldsWrapper {DataFieldValues = list};
    }


    public void Serialize(MongoDB.Bson.IO.BsonWriter bsonWriter, Type nominalType, IBsonSerializationOptions options)
    {
        if (nominalType != typeof (FieldsWrapper))
            throw new ArgumentException("Cannot serialize anything but self");
        bsonWriter.WriteStartDocument();
        foreach (var dataFieldValue in DataFieldValues)
        {

            bsonWriter.WriteName(dataFieldValue.LocalIdentifier);
            if (dataFieldValue.Values.Count != 1)
            {
                var list = new string[dataFieldValue.Values.Count];
                for (int i = 0; i < dataFieldValue.Values.Count; i++)
                    list[i] = dataFieldValue.Values[i];
                BsonSerializer.Serialize(bsonWriter, list); 
            }
            else
            {
                BsonSerializer.Serialize(bsonWriter, dataFieldValue.Values[0]); 
            }
        }
        bsonWriter.WriteEndDocument();
    }

}

There are some other tactics but this seems to be the most applicable one in your case. We tested dictionaries in production and they are as fast as everything else, you will only lose time on mapping if dictionary is not natural for your domain, I would go for custom serialization.

How can I manage changing data structures in MongoDb collections w/ Simple.Data?

1 Answers1

Linked