Issue Deserializing Protobuf files with MonoTouch

Question

We are currently facing the problem that we want to port an application that is currently running on Android (Monodroid) and Wp7 to IOS by using MonoTouch.

That would be not the problem, but deserializing data by using the protobuf-net framework constantly fails with the following exception:

ProtoBuf.ProtoException: Invalid wire-type; this usually means you have over-written a file without truncating or setting the length; see http://stackoverflow.com/q/2152978/23354
  at ProtoBuf.ProtoReader.StartSubItem (ProtoBuf.ProtoReader reader) [0x00000] in <filename unknown>:0
  at ProtoBuf.BclHelpers.ReadGuid (ProtoBuf.ProtoReader source) [0x00000] in <filename unknown>:0
  at TpSerializer.Read (Application.Mobile.TpDataAccess.TimeEntryLoggingDao.Entities.ActiveTimeEntryDbo , ProtoBuf.ProtoReader ) [0x00000] in <filename unknown>:0
  at TpSerializer.Deserialize (Int32 , System.Object , ProtoBuf.ProtoReader ) [0x00000] in <filename unknown>:0
  at ProtoBuf.Meta.TypeModel.TryDeserializeAuxiliaryType (ProtoBuf.ProtoReader reader, DataFormat format, Int32 tag, System.Type type, System.Object& value, Boolean skipOtherFields, Boolean asListItem, Boolean autoCreate, Boolean insideList) [0x00000] in <filename unknown>:0
  at ProtoBuf.Meta.TypeModel.TryDeserializeList (ProtoBuf.Meta.TypeModel model, ProtoBuf.ProtoReader reader, DataFormat format, Int32 tag, System.Type listType, System.Type itemType, System.Object& value) [0x00000] in <filename unknown>:0
  at ProtoBuf.Meta.TypeModel.TryDeserializeAuxiliaryType (ProtoBuf.ProtoReader reader, DataFormat format, Int32 tag, System.Type type, System.Object& value, Boolean skipOtherFields, Boolean asListItem, Boolean autoCreate, Boolean insideList) [0x00000] in <filename unknown>:0
  at ProtoBuf.Meta.TypeModel.DeserializeCore (ProtoBuf.ProtoReader reader, System.Type type, System.Object value, Boolean noAutoCreate) [0x00000] in <filename unknown>:0
  at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream source, System.Object value, System.Type type, ProtoBuf.SerializationContext context) [0x00000] in <filename unknown>:0
  at ProtoBuf.Meta.TypeModel.Deserialize (System.IO.Stream source, System.Object value, System.Type type) [0x00000] in <filename unknown>:0
  at Application.Mobile.TpDataAccess.Core.CoreProtobufDao`2[System.Guid,Application.Mobile.TpDataAccess.TimeEntryDao.Entities.TimeEntryDbo].EnsureCollection () [0x00000] in <filename unknown>:0
  at Application.Mobile.TpDataAccess.TimeEntryDao.TimeEntryProtobufDao.PrepareAccess () [0x00000] in <filename unknown>:0
  at Application.Mobile.TpBusinessComponents.TimeEntryService.TimeEntryService.PrepareAccess () [0x00000] in <filename unknown>:0
  at App.TP.Mobile.AppDelegate.FinishedLaunching (MonoTouch.UIKit.UIApplication app, MonoTouch.Foundation.NSDictionary options) [0x00031] in /Users/Developer/Dropbox/Application Mobile/Application Mobile iOS Project/Application_Mobile/AppDelegate.cs:34
  at (wrapper managed-to-native) MonoTouch.UIKit.UIApplication:UIApplicationMain (int,string[],intptr,intptr)
  at MonoTouch.UIKit.UIApplication.Main (System.String[] args, System.String principalClassName, System.String delegateClassName) [0x0004c] in /Developer/MonoTouch/Source/monotouch/src/UIKit/UIApplication.cs:38
  at App.TP.Mobile.Application.Main (System.String[] args) [0x00000] in /Users/Developer/Dropbox/Application Mobile/Application Mobile iOS Project/Application_Mobile/Main.cs:17

The thing is that we did not change the entities nor the way we are storing the data.

// and serialize
string file = Path.Combine(StoragePath, dataFileName);
using (var stm = new FileStream(file, FileMode.Create))
{
    foreach (var item in list)
        Serializer.Serialize(stm, item);
}

Doesn't matter which entity we read, it fails:

string file = Path.Combine(StoragePath, dataFileName);
if (!File.Exists(file))
    return;

using (var stm = new FileStream(file, FileMode.Open))
{
    var list = (TClass[]) Serializer.Deserialize(stm, null, typeof(TClass[]));
    if (list == null)
        return;

    // transform array into dictionary
    for (int i = 0; i < list.Length; i++)
    {
        var entity = list[i];
        entities[primaryKeyFunction(entity)] = entity;
    }
}

We are using the following Protobuf-NET environment.

Protobuf-net r594 (unity dll)
Pre-created serializer

Saving the data it's not the issue, but the loading fails. Any help is highly appreciated.

Thanks - Gerhard

score 3 · Accepted Answer · answered Oct 18 '12 at 13:33

3

A nice easy one. You have serialized as Foo, and deserialized as Foo[]. Actually, because protocol buffers is an appendable format, by using Serialize repeatedly (to the same file), you are effectively only writing a single Foo (but over-writing most of the members repeatedly).

Basically, to keep your current code, you could use SerializeWithLengthPrefix in place of Serialize, i.e.

foreach(var item in list)
{
    Serializer.SerializeWithLengthPrefix(stm, item,
        PrefixStyle.Base128, Serializer.ListItemTag);
}

but could also have simply used:

Serializer.Serialize(stm, list);

These are both semantically identical; in both cases what you get is (in a dense binary form):

[field 1, string]
[payload length of item 0]
[payload of item 0]
[field 1, string]
[payload length of item 1]
[payload of item 1]
...
[field 1, string]
[payload length of item n]
[payload of item n]

which is exactly what Deserialize is looking for when you tell it TClass[]

answered Oct 18 '12 at 13:33

Marc Gravell

1,026,079
266
2,566
2,900

@BitKFu note that this *by necessity* changes the data on the wire, so it has some compatibility issues, but fundamentally your current `Serialize` loop looks broken for this scenario. – Marc Gravell Oct 18 '12 at 13:37
@BitKFu I would be pretty amazed if that is working *anywhere*. The shape is broken, and there is absolutely no way of getting individual objects back from that use of `Serialize` in a loop... do you have a concrete example? – Marc Gravell Oct 18 '12 at 13:42
@BitKFu can you clarify: what is the type of `item` in the serialize loop? and what is `TClass`? The *only* way I can see it working successfully is if `item` is actually a `List`, i.e. a list-of-lists-of-items, but that loop would flatten it down to a single list-of-items – Marc Gravell Oct 18 '12 at 13:44
Ah Marc, you're right. We changed for IOS, because we missed the proper method overload in the pre-compiled Serializer. original: ProtoBuf.Serializer.Serialize(System.IO.Stream, T) vs. precompiled: ProtoBuf.Meta.TypeModel.Serialize(System.IO.Stream, object) which leaded us to the conclusion that we have to iterate the collection :) – BitKFu Oct 18 '12 at 13:48
@BitKFu so you're all sorted now? – Marc Gravell Oct 18 '12 at 19:24
Think so. But I have to wait for our MonoTouch Developer who can test it. So far, thank you very much for your help. I will approve your answer as soon as I have an update from him. – BitKFu Oct 19 '12 at 06:25
BTW: Which protobuf DLL should be used? Because there's no compilation for MonoTouch, we are currently using the Unity DLLs. – BitKFu Oct 19 '12 at 06:27
@BitKFu /coreonly/ios would be the ideal choice – Marc Gravell Oct 19 '12 at 06:33

Issue Deserializing Protobuf files with MonoTouch

1 Answers1