You looking at a very dark corner. My first attempts at actually testing this functionality foundered on a failure to find a UML tool that could actually draw Fig 7.1. Once I got to grips with the text I realized there are many problems; a potential conflict between two different x[y] syntaxes, further aggravated by a QVTo shortform. There is a problem with extension to more than binary associations. The OCL Abstract Syntax continues to use the UML 1.x concepts of AssociationEnd rather than the UML 2.x Property.
Consequently the functionality you observe is the result of the best endeavours of some tool vendor to make sense of a very inadequate specification.
With MagicDraw, you used to be using Dresden OCL, but I understand that MagicDraw has switched to Eclipse OCL, perhaps the Classic Eclipse OCL. NoMagic seem remarkably reticent in crediting the open source software they redistribute.
For the newer Pivot Eclipse OCL where I prototype solutions to many of the OMG OCL problems, the UML to Pivot loading normalises many of the UML concepts so that Associations are redundant, unless an exoplicit navigation to the Association requires an Association Class to be reified. AssociationClasses are normalized to Association Classes with ordinary Properties for each plausible navigation.
I think that your expression is incorrect.
self.C[b] cannot be a qualified association since the implicit A::C property has no key.
self.C[b] could be a disambiguated navigation A::C where the ambiguity of A::C is resolved by selecting the C::b opposite. But A::C is not ambiguous and its opposite is C::a. So self.C should be adequate, self.C[a] is redundant, self.C[b] is wrong wrt the b. Sadly your tooling dislikes self.C so your tooling is defective.
I think you should have written self.C.val.mod(2) = 0.
Uppercase C is correct. Lowercase was misguidedly suggested by OCL <= 2.2 following the UML style guide. See indented paragraph under "Missing Association End names" in 7.5.4 of OCL 2.4.