0

In an old program I serialized a data structure to bytes, by allocating an array of unsigned char, and then converted ints by:

*((*int)p) = value;

(where p is the unsigned char*, and value is the value to be stored).

This worked fine, except when compiled on Sparc where it triggered exceptions due to accessing memory with improper alignment. Which made perfect sense because the data elements had varying sizes so p quickly became unaligned, and triggered the error when used to store an int value, where the underlying Sparc instructions require alignment.

This was quickly fixed (by writing out the value to the char-array byte-by-byte). But I'm a bit concerned about this because I've used this construction in many programs over the years without issue. But clearly I'm violating some C rule (strict aliasing?) and whereas this case was easily discovered, maybe the violations can cause other types of undefined behavior that is more subtle due to optimizing compilers etc. I'm also a bit puzzled because I believe I've seen constructions like this in lot of C code over the years. I'm thinking of hardware drivers that describe the data-structure exchanged by the hardware as structs (using pack(1) of course), and writing those to h/w registers etc. So it seems to be a common technique.

So my question is, is exactly what rule was violated by the above, and what would be the proper C way to realize the use-case (i.e. serializing data to an array of unsigned char). Of course custom serialization functions can be written for all functions to write it out byte-by-byte but it sounds cumbersome and not very efficient.

Finally, can ill effects (outside of alignment problems etc.) in general be expected through violation of this aliasing rule?

melpomene
  • 84,125
  • 8
  • 85
  • 148
Morty
  • 1,706
  • 1
  • 12
  • 25
  • Its machine specific. E.g. on 68K cpus, you could run into the same problems back in the day - the targets compiler could compensate for that but older compilers and maybe contemporary compilers wont. x86 is the most forgiving type ov CPUs regarding misalignment. You only pay with some extra clock cycles sometimes – BitTickler Sep 28 '15 at 20:36
  • 2
    "Proper" C would be `memcpy(p, &value, sizeof (int));`. – melpomene Sep 28 '15 at 20:39
  • Simple answer: **do not** use proper serialisation instead (bitshifts and bit-operators). There is littel to none gained by relying on _implementation defined behaviour_ (at best) or _undefined behaviour_ (at worst). – too honest for this site Sep 28 '15 at 20:39
  • The proper "way" to serialize a struct is to memcopy it to a byte array of sufficient size. Usually sizeof(struct) >= sum_of_member_sizes(). – BitTickler Sep 28 '15 at 20:41
  • OK but in general, if you're doing something like this that violates the aliasing rules, doesn't the compiler have the right to basically make your program do anything it pleases? I agree that in practice the most obvious problems is with violating the underlying alignment rules on certain architectures, but could there be other consequences? – Morty Sep 28 '15 at 20:42
  • melpomene: OK I agree that it will probably solve the issue since memcpy is alignment-aware. But still doesn't this construction formally break the same aliasing rules that I'm breaking? – Morty Sep 28 '15 at 20:43
  • As soon as you assume some alignment for your implementation, the consequence is that this is quite reliably non-portable code. If you want to "serialize cross platform", you need a proper approach, such as for example google protocol buffers. – BitTickler Sep 28 '15 at 20:44
  • The consequence is always undefined behavior. The compiler is free to exploit it to any extent it pleases its “optimizations”. Use of `memcpy` per the standard does not break aliasing rules. – 5gon12eder Sep 28 '15 at 20:44
  • Sorry, that was badly worded. I'll try again: Simple answer: **do not!** Use proper serialisation instead (bitshifts and bit-operators). There is little to none gained by relying on _implementation defined_ (at best) or even _undefined_ (at worst) _behaviour_. – too honest for this site Sep 28 '15 at 20:47
  • @BitTickler:No, that is not the **proper** way, but the **wrong** way. It relies on implementation details at best, e.g. alignment, endianess, padding, etc. – too honest for this site Sep 28 '15 at 20:48
  • @Morty: Where do you get that `memcpy` is awar of alignment? This is so very wrong! It does not care about alignment, etc, but copies bytes. – too honest for this site Sep 28 '15 at 20:49
  • @Olaf lol - what makes you say that? Did you ever use gpb? The generated C code is portable and works without assuming alignment. As for memcopy of a whole struct, it is just as legal as can be. There is NO system I know of which is not allocating a byte array at "bad alignment" addresses (typically multples of 4 or 8 depending on platform.) – BitTickler Sep 28 '15 at 20:49
  • 5gon12eder: Cool - do you have a reference for that (that use of memcpy per the standard does not break aliasing rules)? I'm also wondering if similar "exceptions" might exist for other functions etc. – Morty Sep 28 '15 at 20:52
  • @BitTickler: Experience,which you seem to be lacking. I presume GPB is google protocol buffers? These do not use simple typecasting of a struct, but uses a meta-language which generates the data corresponding to the actual implementation. This is far from what you recommended. Did you ever look at a PCS and/or ABI? I'm not talking about missalignment of bytes, but the structure fields. – too honest for this site Sep 28 '15 at 20:54
  • @Olaf more LOL - I used it since years now and they do element wise serialization. I pointed out that memcpy of whole structs is working (legal) and that gpb is a proper solution where he can even re-use serialized data cross platform. Those two statements were disjoint, disjunct, unrelated. – BitTickler Sep 28 '15 at 20:55
  • 1
    Olaf: What I meant is that memcpy() is aware of alignment restrictions in the underlying architecture. So if running on an arch where access to, say, 32-bit values has to be to a 32-bit aligned address, it will either copy it byte-by-byte, or it will maybe copy some initial bytes that way and then when the pointers for the remainder of the data is aligned it can copy that in 32-bit chunks etc. Unlike *((*int)p) = v; where an alignment check on p for every such access would be expensive (and thus not required of the compiler). – Morty Sep 28 '15 at 20:55
  • No, it is **not**! @BitTickler seems to hide ignorance under arrogance. If you serialize a struct like that, you copy padding bytes which are (possibly) inserted to align the fields properly, the fields can be big or little endian, etc. `memcpy` is not aware about such issues and a different architecture can have very different demands (not to talk about varying size of standard datatypes and the representation of signed integers or floating point. – too honest for this site Sep 28 '15 at 20:59
  • Omg what is your problem? Can't you treat 2 unrelated statements as 2 statements? They are different posts, too. So what exactly do you imply? – BitTickler Sep 28 '15 at 21:00
  • The C standard does not enforce such a behaviour of `memcpy`. It just has to copy the number of bytes from start to end. It has also no idea about the internal structure of a compound datatype. – too honest for this site Sep 28 '15 at 21:01
  • @BitTickler: I'm refering to "The proper "way" to serialize a struct is to memcopy it to a byte array of sufficient size". As you later came up with GPB, you contradict this statement yourself. – too honest for this site Sep 28 '15 at 21:07
  • `memcpy` is the right solution. The important thing is that you have to `memcpy` it back to a variable of the original type before trying to use it for anything other than opaque data. – Barmar Sep 28 '15 at 21:08
  • The C11 standard (the final draft N1570, actually) says (in § 7.24.2.1 ¶ 2): *The `memcpy` function copies `n` characters from the object pointed to by `s2` into the object pointed to by `s1`. If copying takes place between objects that overlap, the behavior is undefined.* – This is all that is required. As long as your objects are valid and don't overlap, the behavior is well-defined. An implementation is not allowed to impose additional restrictions on a standard library function. – 5gon12eder Sep 28 '15 at 21:08
  • @5gon12eder: The Effective Type rules elsewhere in the standard do impose some absurd additional restrictions which are seldom likely to improve optimization, but do make it absurdly difficult to do certain things in a way which would not justify arbitrary behavior by a conforming-but-capricious implementation. – supercat Feb 23 '17 at 00:38
  • @supercat Do you mean that *implementing* `memcpy` in a way that is efficient *and* standards-compliant is difficult? That's for certain. But *using* it? Fortunately, standard library implementers have more freedom than "ordinary" programmers and can bend the rules a bit if they know that their implementation can cope with it. – 5gon12eder Feb 23 '17 at 18:43
  • @5gon12eder: The rules create corner cases where using it is also difficult. For example, on a system where "int" and "long" have matching 32-bit representations, given a pointer to two allocated buffers of the same size, the first of which may have been written using either `int*` or `long*`, but code doesn't know which, and the other of which may be read using either `int*` or `long*`, copy the values from the first to the second. Under C89 that would have been easy using `memcpy`, but under C99 if code uses `memcpy` and an item from the first buffer was written using `int*`, ... – supercat Feb 23 '17 at 19:14
  • @5gon12eder: ...an attempt to read the corresponding item from the second buffer using `long*` will yield UB because that items Effective Type will have been set based upon the Effective Type in the first buffer. Given that some compilers aggressively assume an `int*` won't alias a `long*` even when `int` and `long` have the same representation, those rules for `memcpy` make an operation which should be trivial (copy all the bytes from one array to another) absurdly complicated. – supercat Feb 23 '17 at 19:19
  • @supercat I used to believe that you can `memcpy` anything into an `int` and afterwards treat it as an `int`. I'll do some research on that and eventually ask a question. This comment thread is probably not the right place to discuss this. – 5gon12eder Feb 23 '17 at 23:17
  • @5gon12eder: For C11, read N1570 6.5 "Expressions" p6, third sentence (it starts with "If a value is copied into an object..."). Copying into an object with a declared type of `int` is no problem--the difficulty lies with copying to allocated storage (which has no declared type) which might next be read by a type which differs from the source. – supercat Feb 24 '17 at 00:41

2 Answers2

1

Yes, your code violates strict aliasing rule. In C, only char* and its signed and unsigned counterparts are assumed to alias other types.

So, the proper way to do such raw serialization is to create an array on ints, and then treat it as unsigned char buffer.

int arr[] = { 1, 2, 3, 4, 5 };
unsigned char* rawData = (unsigned char*)arr;

You can memcpy, fwrite, or do other serialization of rawData, and it is absolutely valid.

Deserialization code may look like this:

int* arr = (int*)calloc(5, sizeof(int));
memcpy(arr, rawData, 5 * sizeof(int));

Sure, you should care of endianness, padding and other issues to implement reliable serialization.

Community
  • 1
  • 1
Stas
  • 11,571
  • 9
  • 40
  • 58
  • It would be equally valid to `memcpy`, `fwrite`, etc the `int arr[]` directly because – as you say – `char *` is assumed to alias, so all raw data handling functions from the standard library may be called with arbitrary objects. There is also no need to cast the arguments of `memcpy` since any pointer implicitly converts to `void *`. – 5gon12eder Sep 28 '15 at 21:59
  • @5gon12eder yes, you are right! But, we don't know how he is going to serialize a buffer (maybe by `putc` :), so casting to `unsigned char*` is needed. Concerning deserialization, yes, no need in casting there. Fixed. Thanks! – Stas Sep 28 '15 at 22:35
0

It is compiler and platform specific, on how a struct is represented (layed out) in memory and whether or not the start address of a struct is aligned to a 1,2,4,8,... byte boundary. Therefore, you should not take any assumptions on the layout of your structs members.

On platforms, where your member types require specific alignment, padding bytes are added to the struct (which equals the statement I made above, that sizeof(struct Foo) >= the sum of its data member sizes). The padding...

Now, if you fwrite() or memcpy() a struct from one instance to another, on the same machine with the same compiler and settings (e.g. in the same program of yours), you will write both the data content and the padding bytes, added by the compiler. As long as you handle the whole struct, you can successfully round trip (as long as there are no pointer members inside the struct, at least).

What you cannot assume is, that you can cast smaller types (e.g. unsigned char ) to "larger types" (e.g. unsigned int) and memcpy between those in that direction, because unsigned int might require proper alignment on that target platform. Usually if you do that wrong, you see bus errors or alike.

malloc() in the most general case is the generic way to get heap-memory for any type of data. Be it a byte array or some struct, independent of its alignment requirements. There is no system existing, where you cannot struct Foo *ps = malloc(sizeof(struct Foo)). On platforms, where alignment is vital, malloc will not return unaligned addresses as it would break any code, trying to allocate memory for a struct. As malloc() is not psychic, it will also return "struct compatible aligned" pointers if you use it to allocate byte arrays.

Any form of "ad hoc" serialization like writing the whole struct is only a promising approach as long as you need not exchange the serialized data with other machines or other applications (or future versions of the same application where someone might have tinkered with compiler settings, related to alignment).

If you look for a portable and more reliable and robust solution, you should consider using one of the main stream serialization packages, one of which being the aforementioned Google protocol buffers.

BitTickler
  • 10,905
  • 5
  • 32
  • 53