Can ASN.1 support putting field lengths in a different place?

Question

I have a protocol whose on-the-wire format is already defined and I'd like to use ASN.1 to encode/decode it, but it seems to break the defined BER/DER/PER options. For whatever reason, the protocol developers did not put payload sizes/lengths immediately before the payload itself - so I can't use the automatic BER/DER. But since the payload can be variable length, I can't use PER either. Here's an example:

     12'b           12'b       4'b     4'b   
|------------|--------------|-------|------|
| some stuff | payload size | blah2 | blah |    Header
|------------------------------------------|
|              payload word 1              |
|------------------------------------------|
|                    ...                   |    Payload
|------------------------------------------|
|              payload word N              |
|------------------------------------------|
| much stuff | many bits | such doge | wow |    Trailer
|------------------------------------------|

So maybe two questions here:

1. Is there a way using one of the ASN.1 encodings to specify certain fields as the length for a later field - so you could say something like bits 9-20 contain the length for bits 33-N*32, but you're skipping bits 21-32 which can have other unrelated junk in them?
2. I can see how an algorithm / rule could be written to support the above, so if there isn't a way to currently do this with ASN.1, is there a way (and documentation) on how to write a new rule or extension of some kind to the existing encodings?

EDIT

To clarify why I'm bringing up ASN.1, without repeating a previous question, is because it's almost exactly what I'm looking for - just apparently without a way to handle the particular use case I'm asking about here. I need to deserialize existing binary protocols and I'd rather not write my own since there are already many tools claiming they can do some form of this. If someone has another suggestion I'd gladly try it.

Answer 1

One of the ASN.1 standards documents specifies something called Encoding Control Notation (ECN). It is made for the purpose of making it possible to use ASN.1 plus the ECN to deal with non-ASN.1 messages. It might do what you need. However, I will warn you that it is pretty complex to work with and I have only seen one company claim to support it (I have no idea how complete their support is).

Answer 2

I believe you can't change the position of the length field inside BER/DER/CER serialization -- it can only be present right before the payload or it can be just a pair of sentinels framing the payload (e.g. indefinite length encoding).

Your ultimate goal is not quite clear, though. Why do you need ASN.1 at all?

Answer 3

I think you may be slightly misunderstanding the goal of ASN.1 – the BER/DER/CER encodings are somewhat auxiliary to ASN.1, even though they're probably much the best known part of it.

ASN.1 is a way of specifying an abstract syntax: it's a way of saying things like ‘message type 13 is a sequence of integer-string pairs of arbitrary length’, without saying how that message is represented. Thus it's part of a system specification, and thus really in the same part of the IT mind-space as other much later methodologies such as UML. From the summary section of X.680:

The ASN.1 notations can be applied whenever it is necessary to define the abstract syntax of information without constraining in any way how the information is encoded for transmission.

The stress is on abstract syntax.

The ‘encoding rules’ are a concrete realisation of such a syntax, defined in a separate standard, X.690, which summarises itself as:

This Recommendation | International Standard defines a set of Basic Encoding Rules (BER) that may be applied to values of types defined using the ASN.1 notation. Application of these encoding rules produces a transfer syntax for such values. [emphasis mine]

In other words, the encoding rules are an example of how one might transfer messages defined using the X.680 syntax, but they are not intended as the only way to do this. The way in which these two standards were originally expected to be used is, I think, that you develop or receive a specification written in ASN.1 syntax, and then use an ‘ASN.1 compiler’ to turn that into a bit of code that can serialise and deserialise that specific syntax. (Despite this, I would guess that the most widely distributed DER parsers, namely those used for X.509 certificates, are generally custom-written rather than derived in this way).

By ‘example’ I mean that the standard is saying that ‘if you have no particular reason to choose another transfer syntax, here's one pre-specified which you might consider’.

That means that the BER/DER/CER/PER encodings are not intended as a completely general way of hauling binary data about, or as a way of expressing the structure of a random wire format, so they're not designed to be extensible in the way your question suggests you wish.

Alternatively, if it had happened that the designers of the wire format you're dealing with had made the same decision as the BER encoding designers, of a strict type-length-payload layout for the format, then you could potentially have reverse-engineered an ASN.1 specification for it, and so used an ASN.1 compiler to generate a (de-)serialiser (hooray, problem solved, knock off early for beers!).

But it looks like they didn't, so you can't unscramble that egg, and I think you're stuck with writing a (de-)serialiser for your wire format by hand. Which means (hooray?) you've got an entertainingly fiddly, but separate, problem to address.