1

Edit

After digging deeper, I learned that my problem is caused by an improperly truncated protobuf stream (once again). I thus reformulated this question to focus on aspects that have not already been answered in the linked questions (see "Refined question").

Background

So, I've inherited this large set of applications that use protobuf-net for just about everything (which is not a bad thing :-)). Customers have tons of serialized data lying around, so I'm bound to staying backwards compatible with the protobuf-net version these applications are currently using (at the very least, I have to be able deserialize the data they have). And here comes the catch: To this date, all parts of the application are based on protobuf-net 1.0.0.282, and I dearly want/need to upgrade. And, as you can imagine, my initial attempts at swapping in newer versions (starting with v2) failed miserably (I get various exceptions, like "Invalid field in source data: 0").

Original question

Staying away from getting into the details of the individual exceptions, is my goal of upgrading to non-primeval versions of protobuf-net while keeping backwards compatibility even feasible? If yes, what would be a good starting point? Are there any resources on how to start such an undertaking? I only found this document in the github repository, but I did not achieve much playing around with the mentioned CompatibilityLevel.

The question(s)

Note: This part of the question has already been answered in the comments - yes, the protocol is compatible Staying away from getting into the details of the individual exceptions, is my goal of upgrading to non-primeval versions of protobuf-net while keeping backwards compatibility even feasible? If yes, what would be a good starting point? Are there any resources on how to start such an undertaking? I only found this document in the github repository, but I did not achieve much playing around with the mentioned CompatibilityLevel.

Refined question

Turns out my real problem is that the original author of the coded decided to serialize the protobuf data into files that are zero-padded at the end. Don't ask me why, but there's nothing I can do about it - this data is living on customer machines. Protobuf-net 1.0.0.282 is perfectly happy with these files, and deserializes them correctly, whereas newer versions (correctly) barf on them. Now, is there anything I can do to make newer versions of protobuf accept the zero-padded files? How can I update without breaking my customers data archives?

bigge
  • 1,488
  • 15
  • 27
  • This is probably too in-depth for a single answer, however maybe the author could point you in an appropriate direction... Paging @MarcGravell (this may or may not work :) ) – TheGeneral Mar 03 '21 at 08:46
  • A good start could be to create various test-objects with the current version, either for existing types, or with types just for testing, or both. And unit tests to de-serialize said objects. This might help with identifying and resolving problems when upgrading. – JonasH Mar 03 '21 at 09:50
  • @TheGeneral (raises gaze... but no: @-ing someone who isn't already involved: does nothing) – Marc Gravell Mar 03 '21 at 09:56
  • @MarcGravell but... you turned up.... so by programming logic, i tried it once, and it worked, so it must work forever (with 0 allocations).... – TheGeneral Mar 03 '21 at 10:41
  • Trying to summon @MarcGravell again - just in case there was no notification about my changes – bigge Jun 11 '21 at 06:15
  • @bigge no, I don't get automatic updates of edits; zero padded files - oof; let me think while I walk the dog – Marc Gravell Jun 11 '21 at 06:41
  • @bigge what library version are you using? I can potentially add something here to make this work, but that would be v3 only – Marc Gravell Jun 11 '21 at 07:21
  • 1
    my idea: https://github.com/protobuf-net/protobuf-net/pull/800 – Marc Gravell Jun 11 '21 at 07:50
  • Great, thanks! I'm trying to update to the latest & greatest, so v3 is more than fine. I'll try it out on Monday when I'm back at work (and have access to some real data). In any case, you deserve more than just a coffee :) – bigge Jun 11 '21 at 18:21
  • Tried out your solution and it works nicely. Thanks! What would we be the way forward from here? – bigge Jun 15 '21 at 04:38
  • @MarcGravell, are you still in the loop? – bigge Jul 07 '21 at 05:00
  • @biggie if you mean about merging that PR: will do; I'm not at a computer for a few days, though. – Marc Gravell Jul 07 '21 at 06:15
  • Ok, thanks. Didn't want to rush, was just worried because I never got feedback for my corresponding comment to the pull request – bigge Jul 08 '21 at 05:59

1 Answers1

1

As far as I can recall, there were no fundamental configuration changes around the time of 1.x to 2.x (which was a very long time ago), so this should - as far as I know - be a very low effort upgrade. If something unexpected is happening, an example type and payload would be useful, although this may be a better topic for GitHub than Stack Overflow.

The actual data protocol hasn't changed at any point, so there is no fundamental reason this shouldn't work, and to be honest I'm surprised you've hit any snags at all. I'd be happy to help further, but: I'd really need to see the problem in front of me.

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
  • Thanks for the immediate reply! As mentioned above, I tried, failed miserably, and started googling for upgrade/migration guidelines, which I could not find (except for the github link above). As a result, I posted this question as a place to gather such information. Now that I know that it _should_ just be working, I have enough confidence to dig into the details of what is failing in my particular case. I will update then update my question with details (and potentially open a github issue). – bigge Mar 03 '21 at 13:57
  • It's been way too long, but I've finally been able to work on this again. Please see my updates to the question. Thanks! – bigge Jun 09 '21 at 05:27