c++ binary data layout guaranteed by the standard

Question

This is purely a theoretical problem, nothing I have really found myself in, but it has piqued my curiosity and wanted to see if anyone has a better solution for it:

How do you portably guarantee that an specific file format / network protocol or whatever conforms to a specific bit pattern.

Say we have a file format that uses a 64 bit header struct immediately followed by a variable length array of 32 bit structures:

Header:  magic : 32 bit
         count : 32 bit

Field :  id   : 16 bit
         data : 16 bit

My first instinct would be to write something like:

struct Field
{
    uint16_t id   ;
    uint16_t data ;
};

Except that our compiler may decide that padding is advisable and we end up with a 64 bit structure. So our next bet is:

using Field = uint16_t[2];

and work on that.

That is, unless someone has carefully read the standard and noticed that uint16_t is optional. At this point our next best friend is uint_least16_t, which is guaranteed to be at least 16 bits long, but for all we know could be 20 bits long in a 10 bit / char processor.

At this point, the only real solution I can come up with is some sort of bit stream, capable of reading and writing specific amounts of bits, and adaptable by std::numeric_limits.

So, is there someone out there who has very carefully read the standard and found the point I'm missing? Or it is this the only real way of having a portable guarantee.

Notes: - I've just realized that endianness would probably add another layer of complexity. - I'm using the current working draft of the ISO standard (N3797).

*Header: magic : 32 bit* ... *uint16_t magic;* -- clearly not guaranteed — David Rodríguez - dribeas, Dec 10 '13 at 14:07
Universal portability is a pipe dream. Pick a finite set of platforms you are interested in, and write for them. — Benjamin Lindley, Dec 10 '13 at 14:22
You're missing the forest for the trees. Consider *why* `uint16_t` is optional (the Standard can't assume every platform can provide it), and how that impacts your implied assertion that *all* code should be 100% Standards-conformant. — John Dibling, Dec 10 '13 at 14:38
This is just a theoretical question. I'm pulling at the tail of the standard and yes, it bites back!! — Diego Sánchez, Dec 10 '13 at 16:12

score 2 · Answer 1 · answered Dec 10 '13 at 14:04

2

If you want to ensure portability to everything standard-conforming, including platforms for which CHAR_BITS isn't 8, well, you've got your work cut out for you.

If you are comfortable limiting yourself to 98% of the computers you'll ever program, I recommend writing explicit serialization for anything that has to adhere to a particular wire-format. That includes breaking integers into bytes, etc.

Write appropriate abstractions around things and the code won't be too bad. Don't put shifts and masks everywhere. Encapsulate it.

answered Dec 10 '13 at 14:04

Joe Z

17,413
3
28
39

1

98% is a severe under-estimate. – Benjamin Lindley Dec 10 '13 at 14:06
@BenjaminLindley : I guess it depends on if you ever end up doing embedded programming. I program at least one machine regularly which would want CHAR_BITS = 16. – Joe Z Dec 10 '13 at 14:07

John Dibling · Accepted Answer · 2013-12-10T14:40:03.633

How do you portably guarantee that an specific file format / network protocol or whatever conforms to a specific bit pattern.

You can't. Not in C++, which was standardized against an abstract platform where little more than the existence of a "byte" that is made up of bits can be assumed. We can't even say for certain, in looking only at the Standard, how many bits are in a char. You can use bitfields for everything, as bits are indivsible, but then you'll have padding to contend with at the least.

Sometimes it is best to give up on the idea of absolute Standards conformance for the sake of conformance, and look to other means to get the job done efficiently and effectively. In this case, platform specifics in combination with almost absolute Standards conformance (aka, good programming practices) will set you free.

Every platform I work on regularly (linux & windows) provides a means to regulate the padding the compiler will actually apply. For network communications, under Linux & Windows I use:

#pragma pack (push, 1)

as a preface to all the data structures I'm going to send over the wire. Endianness is indeed another challenge, but one more or less easily dealt with using other resources provided by every platform: ntohl and the like.

Standards conformance is a laudable goal, and indeed in a code review I would reject most code that is non-conformant. The lack of conformance is really just a moniker for the rejection however; not the reason itself. The actual reason for the rejection is in large part difficulty in maintaining and porting non-conformant code when moving to another platform, or indeed even just upgrading the compiler on the same platform. Non-conformant code might compile and even appear to work, but it will very often fail in subtle and miserable ways when you least expect it, even after thorough testing.

The moral of the story is:

You should always write Standards-conformant code, except when you shouldn't.

This really is just a re-imagining of Einstein's articulation of Occam's Razor:

Make everything as simple as possible, but no simpler.

Sorry, no. Just forgot to. Solved. In fact, I believe it to be a really good answer. — Diego Sánchez, Dec 10 '13 at 16:23

score 0 · Answer 3 · edited May 23 '17 at 11:57

0

I would use network types and network byte orders. See this link.http://www.beej.us/guide/bgnet/output/html/multipage/htonsman.html. The example uses uint16_t. You can write the values a field at a time to prevent padding. Or if you want to read and write the entire structure at one see this link C++ struct alignment question

edited May 23 '17 at 11:57

Community

1
1

answered Dec 10 '13 at 14:34

Robert Jacobs

3,266
1
20
30

score 0 · Answer 4 · answered Dec 10 '13 at 15:02

Make the structure easy for the program to use.

Provide input methods that extract data from the input and write to the data members. This removes the issue of padding, alignment boundaries and endianness. Similarly with output.

For example, if your input data is 16-bits wide, but your platform is 32-bits wide, declare the structure using 32-bit fields. Copy the 16 bits from the input into the 32-bit fields.

Most programs read into a structure fewer times than they access the data members. Your program is not reading the input 100% of the time.

c++ binary data layout guaranteed by the standard

4 Answers4