23

I looked at this answer and I am in a situation where I don't need to maintain backward compatibility and I have to have a solution that works without having to decorate dozens of classes with the attributes needed for protobuf-net. So I tried using RuntimeTypeModel.Default.InferTagFromNameDefault = true; but I may be not using it correctly because the Serializer.Serialize call still throws an exception asking for a contract. Here is my quick test, what am I doing wrong?

public enum CompanyTypes
{
    None, Small, Big, Enterprise, Startup
}

public class BaseUser
{
    public string SSN { get; set; }    
}

public class User : BaseUser
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }
    public DateTime BirthDate { get; set; }
    public List<string> Friends { get; set; }
    public Company Company { get; set; }
}

public class Company
{
    public string Name { get; set; }
    public string Address { get; set; }
    public CompanyTypes Type { get; set; }
    public List<Product> Products { get; set; }
}

public class Product
{
    public string Name { get; set; }
    public string Sku { get; set; }
}

[TestClass]
public class SerializationTest
{
    [TestMethod]
    public void SerializeDeserializeTest()
    {
        var user = new User
                       {
                           Age = 10,
                           BirthDate = DateTime.Now.AddYears(-10),
                           FirstName = "Test First",
                           LastName = "Test Last",
                           Friends = new List<string> { "Bob", "John" },
                           Company = new Company
                                         {
                                             Name = "Test Company",
                                             Address = "Timbuktu",
                                             Type = CompanyTypes.Startup,
                                             Products = new List<Product>
                                             {
                                                new Product{Name="Nerf Rocket", Sku="12324AC"},
                                                new Product{Name="Nerf Dart", Sku="DHSN123"}
                                             }
                                         }
                       };

        RuntimeTypeModel.Default.InferTagFromNameDefault = true;
        using (var memoryStream = new MemoryStream())
        {
            Serializer.Serialize(memoryStream, user);
            var serialized = Convert.ToBase64String(memoryStream.GetBuffer(), 0, (int)memoryStream.Length);
        }
    }
}
Community
  • 1
  • 1
codelove
  • 1,396
  • 3
  • 17
  • 35

3 Answers3

29

InferTagFromName (and it's twin, InferTagFromNameDefault) only take a hand when it is necessary to resolve a tag number for a member; they don't influence which members need to be serialized (so currently the answer to that would be: none, even if the system knew about them). The option you might have chosen would be ImplicitFields, but that is currently only available as a [ProtoContract(...)] marker. If you don't mind a little annotation, a pragmatic fix may be:

[ProtoContract(ImplicitFields = ImplicitFields.AllPublic)]

on User, Company and Product, and something a bit more complex for BaseUser (because of the inheritance):

[ProtoContract(ImplicitFields = ImplicitFields.AllPublic, ImplicitFirstTag = 10)]
[ProtoInclude(1, typeof(User))]

Note we haven't had to add lots of per-member annotation. If you are really really anti-attributes, then it is also possible to configure the entire model through code, via:

RuntimeTypeModel.Default.Add(typeof(Product), false).Add("Name", "Sku");
RuntimeTypeModel.Default.Add(typeof(Company), false).Add("Name", "Address",
         "Type", "Products");
RuntimeTypeModel.Default.Add(typeof(User), false).Add("FirstName", "LastName",
         "Age", "BirthDate", "Friends", "Company");
RuntimeTypeModel.Default.Add(typeof(BaseUser), false).Add(10, "SSN")
         .AddSubType(1, typeof(User));
Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • 4
    Thanks for the detailed reply, Marc. Unfortunately the minimal annotation or the RuntimeTypeModel route won't work for us because our framework is used by developers inside and outside the company and we don't want them to have to reference Protobuf assemblies and have to decorate their inherited classes. One option is to use reflection to spool up RuntimeTypeModel, which is not necessarily trivial so for now I am falling back to the good old (and slow) BinaryFormatter. Do you have any plans to make Protobuf work without any annotation? Thanks again for your work on Protobuf, it is awesome! – codelove Sep 07 '12 at 13:34
  • 4
    @codelove what I'd like to do is add an event to the model that is triggered when a new type is found, allowing your code to react and configure it *as discovered* - would that help? What I also need to do is make the "implicit fields" behaviour accessible via the model rather than just via attributes. – Marc Gravell Sep 07 '12 at 13:45
  • Marc: Yes, an event for each new type discovered makes it a little bit easier, but I would still have to have some pipeline that would pre load or lazy load all types and their properties and fields through reflection and push them into the model. So this event by itself won't help much. However, combined with your other suggestion to allow "implicit fields" via model would. Because then I can subscribe to that event and as each type is discovered I tell it to pickup fields implicitly. This approach allows me not have to reflect on each type. – codelove Sep 07 '12 at 19:23
  • Marc: I am not sure if you are the only person working on this project but if you need a hand, I'd be happy to add this piece. – codelove Sep 07 '12 at 19:31
  • @MarcGravell I am having similar trouble in this scenario: https://stackoverflow.com/questions/57116127/object-fingerprinting-serialization-untouchable-legacy-code-getter-only-aut – alelom Jul 19 '19 at 17:03
  • @MarcGravell thank you for taking a notable role in the community. I had been searching for a way to do this for a couple hours; trying to avoid using method I figured would work. It did not... Then, a couple more hours of trying stuff, and then seeing this post. RuntimeTypeModel.Default.Add will allow for me to serialize existing 3rd party (microsoft) objects/classes. This approach can be put in a single utility function and require less nonsense. I was a hour away from just saying f*** it, and reverting it back to a REST API, and doing a gRPC service another day. Thanks much. – fafafooey Oct 12 '21 at 15:47
5

The question is old, but maybe someone will need this. I implemented ProtobufSerializer class which will construct your Type graph upon use. You just need to annotate your DTO with [KnownTypeAttribute] and [DataMember]/[IgnoreDataMember] attributes. Mostly this is refactored version of another nuget project from some guy. This way you don't need to include protobuf in your contract dependencies:

    internal sealed class ProtobufSerializer
    {
        private readonly RuntimeTypeModel _model;
        private const BindingFlags Flags = BindingFlags.FlattenHierarchy | BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance;
        private readonly Dictionary<Type, HashSet<Type>> _subTypes = new Dictionary<Type, HashSet<Type>>();
        private readonly ConcurrentDictionary<Type, bool> _builtTypes = new ConcurrentDictionary<Type, bool>();
        private static readonly Type[] ComplexPrimitives = new [] { typeof(object), typeof(ValueType), typeof(Enum), typeof(Array)};
        private readonly object _sync = new object();

        public ProtobufSerializer()
        {
            _model = TypeModel.Create();
        }

        public void Serialize(Stream s, object input)
        {
            EnsureType(input.GetType());
            _model.Serialize(s, input);
        }

        public T Deserialize<T>(Stream s)
        {
            EnsureType(typeof(T));
            return (T)_model.Deserialize(s, null, typeof(T));
        }

        public void EnsureType(Type type)
        {
            if (_builtTypes.ContainsKey(type))
            {
                return;
            }
            lock (_sync)
            {
                if (_builtTypes.ContainsKey(type))
                {
                    return;
                }
                var all = GetGraph(type).ToArray();
                foreach (var t in all)
                {
                    InternalBuild(t);
                }
            }
        }

        private void InternalBuild(Type type)
        {
            if (IsPrimitive(type))
            {
                return;
            }

            FlatBuild(type);
            EnsureBaseClasses(type);
            EnsureGenerics(type);

            _builtTypes.TryAdd(type, false);
        }

        private bool IsPrimitive(Type type)
        {
            return type == null || type.IsPrimitive || _model.CanSerializeBasicType(type) || _builtTypes.ContainsKey(type) || ComplexPrimitives.Contains(type);
        }

        private static IEnumerable<Type> GetGraph(Type type)
        {
            return type.TraverseDistinct(GetConnections).Distinct().OrderBy(x=> x.FullName);
        }

        private static Type GetParent(Type type)
        {
            return type.BaseType;
        }

        private static IEnumerable<Type> GetChildren(Type type)
        {
            var knownTypes = type.GetCustomAttributes(typeof(KnownTypeAttribute)).Cast<KnownTypeAttribute>().Select(x => x.Type).ToArray();
            foreach (var t in knownTypes)
            {
                yield return t;
            }

            var fields = GetFields(type);
            var props = GetProperties(type);
            foreach (var memberType in fields.Select(f => f.FieldType))
            {
                yield return memberType;
            }
            foreach (var memberType in props.Select(f => f.PropertyType))
            {
                yield return memberType;
            }
        }

        private static IEnumerable<Type> GetConnections(Type type)
        {
            var parent = GetParent(type);
            if (parent != null)
            {
                yield return parent;
            }
            var children = GetChildren(type);
            if (children != null)
            {
                foreach (var c in children)
                {
                    yield return c;
                }
            }
        }

        private void FlatBuild(Type type)
        {
            if(type.IsAbstract)
                return;

            var meta = _model.Add(type, false);
            var fields = GetFields(type);
            var props = GetProperties(type);
            meta.Add(fields.Select(m => m.Name).ToArray());
            meta.Add(props.Select(m => m.Name).ToArray());
            meta.UseConstructor = false;
            foreach (var memberType in fields.Select(f => f.FieldType).Where(t => !t.IsPrimitive))
            {
                InternalBuild(memberType);
            }
            foreach (var memberType in props.Select(f => f.PropertyType).Where(t => !t.IsPrimitive))
            {
                InternalBuild(memberType);
            }
        }

        private static FieldInfo[] GetFields(Type type)
        {
            return type.GetFields(Flags).Where(x => x.IsDefined(typeof(DataMemberAttribute))).Where(x => !x.IsDefined(typeof(IgnoreDataMemberAttribute))).ToArray();
        }

        private static PropertyInfo[] GetProperties(Type type)
        {
            return type.GetProperties(Flags).Where(x => x.IsDefined(typeof(DataMemberAttribute))).Where(x=> !x.IsDefined(typeof(IgnoreDataMemberAttribute))).ToArray();
        }

        private void EnsureBaseClasses(Type type)
        {
            var baseType = type.BaseType;
            var inheritingType = type;


            while (!IsPrimitive(baseType))
            {
                HashSet<Type> baseTypeEntry;

                if (!_subTypes.TryGetValue(baseType, out baseTypeEntry))
                {
                    baseTypeEntry = new HashSet<Type>();
                    _subTypes.Add(baseType, baseTypeEntry);
                }

                if (!baseTypeEntry.Contains(inheritingType))
                {
                    InternalBuild(baseType);
                    _model[baseType].AddSubType(baseTypeEntry.Count + 500, inheritingType);
                    baseTypeEntry.Add(inheritingType);
                }

                inheritingType = baseType;
                baseType = baseType.BaseType;
            }
        }

        private void EnsureGenerics(Type type)
        {
            if (type.IsGenericType || (type.BaseType != null && type.BaseType.IsGenericType))
            {
                var generics = type.IsGenericType ? type.GetGenericArguments() : type.BaseType.GetGenericArguments();

                foreach (var generic in generics)
                {
                    InternalBuild(generic);
                }
            }
        }
    }

And also some simple extensions:

    public static IEnumerable<T> TraverseDistinct<T>(this T enumer, Func<T, IEnumerable<T>> getChildren)
    {
        return new[] { enumer }.TraverseDistinct(getChildren);
    }
    public static IEnumerable<T> TraverseDistinct<T>(this IEnumerable<T> enumer, Func<T, IEnumerable<T>> getChildren)
    {
        HashSet<T> visited = new HashSet<T>();
        Stack<T> stack = new Stack<T>();
        foreach (var e in enumer)
        {
            stack.Push(e);
        }
        while (stack.Count > 0)
        {
            var i = stack.Pop();
            yield return i;
            visited.Add(i);

            var children = getChildren(i);
            if (children != null)
            {
                foreach (var child in children)
                {
                    if (!visited.Contains(child))
                    {
                        stack.Push(child);
                    }
                }
            }
        }
    }
eocron
  • 6,885
  • 1
  • 21
  • 50
3

It's quite experimental at this phase but I've made a small library that takes most Types and generates the Protobuf-net serializers at run-time: https://github.com/fnicollier/AutoProtobuf

Fabian Nicollier
  • 2,811
  • 1
  • 23
  • 19