Automatically check for binary incompatibility between versions of a C# DLL?

Question

I'm looking for any automatable way to check for binary incompatibility between versions of a C# DLL.

Something like Unix C++'s abi-compliance-checker but for .NET, and in particular for C#, would be great. This SO answer gives a screenshot of exactly the kind of info that I'd love to be able to get regarding two different versions of a C# API.

If there isn't any direct equivalent, then it would also be helpful to get any information at all about any automatable way to get some assurance that binary compatibility isn't broken between versions of a library.

This question seems to be the authoritative list on Stack Overflow of ways to break your API. There's also an article by Jon Skeet which discusses some of these issues. (And of course there are multiple other discussions of this issue on the internet; though there is a lot more of it focused on C++ than on C# and .NET, which is what I'm asking about here.)

Amongst other things, those articles help make clear that:

• Binary and compilation compatibility and incompatibility are NOT trivial problems!

• Binary (in)compatibility does not always entail compilation (in)compatibility nor vice versa

  • Even though (it might be worth noting here) at least one Microsoft article on the topic incorrectly/misleadingly states that it does in one direction

• There are different levels of incompatibility: in particular, there are some obscure things, particularly to do with reflection, that users of a library can do which will make certain API changes binary incompatible, where they wouldn't have been otherwise

  • As a side note, this pretty much makes it seem like real world incompatibility isn't a yes/no question: there'll be certain theoretically incompatible changes which you might well be prepared to make within a major version of your API, if you don't expect normal users of the API to be doing these more obscure things with it
  • That fits with the different severities of incompatibility shown in the abi-compliance-checker screenshot in the SO answer linked above

So, given that everybody these days is supposed to be updating their APIs using semantic versioning, and also given that everybody is using test suites and probably devops as well, how on earth is everybody - or even anybody?! - automatically testing for binary incompatibility changes in .NET?

I can see that careful use of a test suite is enough to let you know if you've made compilation compatibility changes, and most projects already have this - hence why my question is about binary incompatibility changes, which don't seem to be automatically detected at all in any projects I've looked at or worked on.

Just reading the above articles and being careful probably isn't enough... (though of course it's some kind of start) but I can't find any documentation about any other approach.

Answer 1

As a partial answer to my own question, and on the basis that something is better than nothing at all, it looks to me as if one possible pragmatic answer (i.e. it definitely won't catch all possible problems, but it should at least catch some...) would be to store compiled versions of old test suites and then to check whether these still run against compiled newer versions of the API.

In essentially all OSS projects I've worked on, the tests and the library are compiled at the same time. Anything which is a binary incompatibility but not a compilation incompatibility would never cause a failure when just running normal tests in the normal way, in such a project. But at least some such problems would show up with what I've suggested.

So I'm not just answering my question by saying 'have a test suite'! I'm suggesting something different from what I've seen normally done with test suites, that can (and does, I now use this approach) help with the problem my question is about.