6

When doing pointer arithmetic with offsetof, is it well defined behavior to take the address of a struct, add the offset of a member to it, and then dereference that address to get to the underlying member?

Consider the following example:

#include <stddef.h>
#include <stdio.h>

typedef struct {
    const char* a;
    const char* b;
} A;

int main() {
    A test[3] = {
        {.a = "Hello", .b = "there."},
        {.a = "How are", .b = "you?"},
        {.a = "I\'m", .b = "fine."}};

    for (size_t i = 0; i < 3; ++i) {
        char* ptr = (char*) &test[i];
        ptr += offsetof(A, b);
        printf("%s\n", *(char**)ptr);
    }
}

This should print "there.", "you?" and "fine." on three consecutive lines, which it currently does with both clang and gcc, as you can verify yourself on wandbox. However, I am unsure whether any of these pointer casts and arithmetic violate some rule which would cause the behavior to become undefined.

curiousguy
  • 8,038
  • 2
  • 40
  • 58
Ben Steffan
  • 1,095
  • 12
  • 16
  • I'm very curious about *why* you want to do something like this? Is it plain curiosity? Or is there some underlying problem that you want to solve this way? If the latter, then perhaps you should ask about *that* instead? – Some programmer dude Oct 02 '17 at 10:55
  • @Someprogrammerdude Mostly curiosity. The idea just popped up when writing code, and it could indeed mean a (very small) optimization, but there is no real problem that I am trying to solve here. I am specifically interested in this `offsetof` use. – Ben Steffan Oct 02 '17 at 11:04
  • 1
    When you mention "optimization" I have alarm bells going of in my head. Don't do *premature optimizations*, instead write simple, readable, and most importantly *maintainable* code first and foremost. Then remember that "good enough" often *is* good enough. And only if the performance of your program isn't "good enough" for your requirements you measure, profile and benchmark to find the bottlenecks, and fix only the *worst* of those (with plenty of comments and documentation). – Some programmer dude Oct 02 '17 at 11:12
  • 2
    You violate the strict aliasing rule. – Klas Lindbäck Oct 02 '17 at 11:13
  • 1
    @Someprogrammerdude I'm well aware of the danger of premature optimization. Maybe I didn't express myself clearly enough, but I do not actually intend to use this unless required by performance deficits etc. This is just the context in which I came up with this idea. I asked this question because I'm genuinely curious about whether this would actually be possible or not, not because I intend to use it. – Ben Steffan Oct 02 '17 at 11:21
  • @BenSteffan I'm not even sure how doing this manually would be any kind of optimization in any case. Using direct member access with `test[i].b` will do the same pointer arithmetic as necessary, but is guaranteed to be correct behavior while pointer hacks are not. And since it's the common case, it's the one that compiler designers would focus optimization on, whereas they might not have as effectively optimized pointer hacks. One could imagine a case where `test[i].b` is compiled to a single `load` instruction while the manual arithmetic is done as individual steps followed by a `load`. – zstewart Oct 03 '17 at 19:16
  • @zstewart: If multiple structures will have certain fields in the same places, and one needs a function that can work with all such structures interchangeably, I'm not sure how one could write the code without using either `offsetof` or a compiler that actually processes the Common Initial Sequence guarantees in usable fashion. – supercat Oct 04 '17 at 14:25
  • @supercat oh, interesting. I'd gotten the wrong impression about the ability to cast between structs with common prefixes because I'd though Python was doing it -- and they *were* [but it looks like they aren't anymore](http://legacy.python.org/dev/peps/pep-3123/). – zstewart Oct 05 '17 at 16:06
  • @zstewart: The ability to cast between structs with common prefixes is allowed by the Standard if a complete union type declaration containing both types is visible when the struct is accessed, but the authors of gcc claim that the rule says that it's only required to honor the CIS rule when all lvalue accesses are performed through the union type, even though that isn't what the rule says and would make the rule useless. – supercat Oct 05 '17 at 17:48

2 Answers2

1

As far as I can tell, it is well-defined behavior. But only because you access the data through a char type. If you had used some other pointer type to access the struct, it would have been a "strict aliasing violation".

Strictly speaking, it is not well-defined to access an array out-of-bounds, but it is well-defined to use a character type pointer to grab any byte out of a struct. By using offsetof you guarantee that this byte is not a padding byte (which could have meant that you would get an indeterminate value).

Note however, that casting away the const qualifier does result in poorly-defined behavior.

EDIT

Similarly, the cast (char**)ptr is an invalid pointer conversion - this alone is undefined behavior as it violates strict aliasing. The variable ptr itself was declared as a char*, so you can't lie to the compiler and say "hey, this is actually a char**", because it is not. This is regardless of what ptr points at.

I believe that the correct code with no poorly-defined behavior would be this:

#include <stddef.h>
#include <stdio.h>
#include <string.h>

typedef struct {
    const char* a;
    const char* b;
} A;

int main() {
    A test[3] = {
        {.a = "Hello", .b = "there."},
        {.a = "How are", .b = "you?"},
        {.a = "I\'m", .b = "fine."}};

    for (size_t i = 0; i < 3; ++i) {
        const char* ptr = (const char*) &test[i];
        ptr += offsetof(A, b);

        /* Extract the const char* from the address that ptr points at,
           and store it inside ptr itself: */
        memmove(&ptr, ptr, sizeof(const char*)); 
        printf("%s\n", ptr);
    }
}
Lundin
  • 195,001
  • 40
  • 254
  • 396
  • "Not sure why you did that cast, since it is superfluous." -- The cast makes perfect sense. `ptr` has type `char *` and points to the first byte of a pointer value. It needs to be cast to a pointer-to-pointer type to access the actual pointer value. You might argue that it should be `const char **` rather than `char **`, but there definitely needs to be a cast there. –  Oct 02 '17 at 11:57
  • @hvd Fair enough, it is not superfluous, it is plain UB. Converting a `char*` to a `char**` is an invalid pointer conversion, even if the effective type of the pointed-at data happened to be a pointer. – Lundin Oct 02 '17 at 12:04
  • 1
    Converting a `char *` to a `char **` is supposed to be valid when the value pointed to really is a `char *`. Otherwise, the whole `offsetof` macro would be pretty difficult to use correctly (as @cmaster points out, it's still possible). Where does the standard say it is an invalid pointer conversion? 6.3.2.3p7 at least says that given `char *p;`, `char *q = (char *) &p;` is valid, and after that, `(char **) q == &p` must hold. –  Oct 02 '17 at 12:17
  • @Lundin So, you think that replacing the cast `(char**)ptr` with the equivalent `char* string; memcpy(&string, ptr, sizeof(string));` would make the code snippet well-defined? – cmaster - reinstate monica Oct 02 '17 at 12:18
  • @hvd 6.3.2.3/7 specifies what conversion is possible as far as the pointer syntax goes - C allows all manner of wild pointer casts. Whether you can actually access that pointed-at data or not is determined by effective type/strict aliasing, 6.5/6 and 6.5/7. And well, the effective type of the object is `const char*`. So I suppose the only issue is the `const` after all. If `ptr` is a pointer to such an object, then the proper type to use would be `char *const *`. While `char**` is an incompatible type. – Lundin Oct 02 '17 at 13:30
  • @cmaster Apart from the const issue, then yes that code would also be well-defined. – Lundin Oct 02 '17 at 13:33
  • Hmm, given some more thought... The issue I wasn't certain about is `ptr` itself, not the pointed at data. `ptr` is clearly a `char*` in the original code, you can't tell the compiler that `ptr` is actually a `char**`. A `char**` cannot alias a `char*` - this is the actual issue! Not the pointed at data. I'll update the answer. – Lundin Oct 02 '17 at 13:40
  • @Lundin Both, really. 6.3.2.3p7 is required to determine whether the converted pointer points to the intended object, 6.5 is then required to determine whether the access to that object is allowed by the aliasing rules. As for your update, "so you can't lie to the compiler and say "hey, this is actually a `char**`"" -- the code isn't doing that. `(char***)&ptr` would be doing that, and that would indeed be invalid, but `(char**)ptr` only converts the value stored in `ptr` to `char**`, it doesn't pretend that `ptr` is a `char**`. –  Oct 02 '17 at 14:31
  • @hvd `ptr` being a `char*`, can only point at characters, not at other pointers. Of course there's a pointer `b` sitting right there in the binary, but the compiler is free to assume that the line`(char**)ptr` is _not_ used to access the pointer `b`, since a `char**` cannot be used to access a `char*`. Regardless of the const issue. – Lundin Oct 02 '17 at 14:40
  • @hvd You can verify that this is truly UB by writing to `b` with this `char**` method on various compilers and then print the value of `b` afterwards. The value might not have been updated, because of the strict aliasing violation. Most likely compiler to misbehave is gcc, because it optimizes such code aggressively. – Lundin Oct 02 '17 at 14:42
  • @Lundin "`ptr` being a `char*`, can only point at characters, not at other pointers." -- There's a special exception for character types. They may be used to point to and access any object. That's what 6.3.2.3p7 covers in its last two sentences, and what the special exception in 6.5p7 ("a character type") is for. And that's well supported even in GCC. –  Oct 02 '17 at 14:47
  • @hvd Yes, I am well aware. You can use it to read out the binary representation of any data type (in which case you'll actually want `unsigned char` or `uint8_t`). But that's not how it was used here! To do that you'd have to create some non-portable monstrosity like `const char* p = ptr[0] << 24 | ptr[1] << 16`, which assumes that you are aware of the specific pointer representation and its endianess. – Lundin Oct 02 '17 at 14:53
  • @Lundin Or, as 6.3.2.3p7 allows in some cases, just cast the `char *` value back to a pointer to the proper type, `const char **` in this case, and dereference that. Which is what the OP is doing. You had *just* claimed it is invalid because `char *` cannot point to pointer objects, only to characters, but you seem to acknowledge now that both pointing to other objects and accessing other objects is allowed. –  Oct 02 '17 at 14:57
  • @hvd The proper type of `ptr` in the question is `char*` and nothing else, period. As I already said, it doesn't matter what it points at, because **the compiler will assume that the pointed-at data, what ever it might be, will not be accessed through a `char**`**. The strict aliasing rule has an exception for the purpose of serialization of binary data byte by byte, and that was the example I just gave since you thought that it applied here - but it has nothing at all to do with the OP's case. – Lundin Oct 02 '17 at 15:09
  • @Lundin No, the compiler will not assume that, because there is no rule anywhere in the standard that allows the compiler to assume that. In the OP's case, there is no access of the `b` member using an lvalue of `char`, because `ptr` is never dereferenced directly, and even if the `b` member *were* dereferenced directly, the exception for character types always applies, because the standard does not and cannot make exceptions depending on what's in the programmer's mind at the time, it cannot possibly only apply when the purpose is serialisation. –  Oct 02 '17 at 15:18
  • @Lundin: Seeing what compilers do is not an effective way of testing whether something is defined by the Standard. Knowing that a compiler handles certain constructs bogus fashion may be useful, but that doesn't imply that the Standard doesn't define those constructs. – supercat Oct 04 '17 at 17:29
1

Given

struct foo {int x, y;} s;
void write_int(int *p, int value) { *p = value; }

nothing in the Standard would distinguish between:

write_int(&s.y, 12); //Just to get 6 characters

and

write_int((int*)(((char*)&s)+offsetof(struct foo,y)), 12);

The Standard could be read in such a way as to imply that both of the above violate the lvalue-type rules since it does not specify that the stored value of a structure can be accessed using an lvalue of a member type, requiring that code wanting to access as structure member be written as:

void write_int(int *p, int value) { memcpy(p, value, sizeof value); }

I personally think that's preposterous; if &s.y can't be used to access an lvalue of type int, why does the & operator yield an int*?

On the other hand, I also don't think it matters what the Standard says. Neither clang nor gcc can be relied upon to correctly handle code that does anything "interesting" with pointers, even in cases that are unambiguously defined by the Standard, except when invoked with -fno-strict-aliasing. Compilers that make any bona fide effort to avoid any incorrect aliasing "optimizations" in cases which would be defined under at least some plausible readings of the Standard will have no trouble handling code that uses offsetof in cases where all accesses that will be done using the pointer (or other pointers derived from it) precede the next access to the object via other means.

GermanNerd
  • 643
  • 5
  • 12
supercat
  • 77,689
  • 9
  • 166
  • 211
  • Would you recommend to *always* use -fno-strict-aliasing? It seems to me to be the only sane way left. And do you know if all current compilers offer a similar flag? If not, what other options are there? – GermanNerd Oct 04 '19 at 06:31
  • @GermanNerd: People seeking to sell compilers design them to fulfill customers' needs. It's possible that some specialized compilers might not offer options to disable obtuse "optimizations" that break commonplace constructs, but any quality compiler that is designed to be suitable for low-level programming will offer a mode that will handle the constructs that gcc/clang optimizers can't. Getting anything useful in the Standard would require a consensus among three factions: those who want to allow aggressive optimizations by implementations intended for purposes where they would be fine,... – supercat Oct 04 '19 at 15:43
  • ...those who want to be able to use low-level constructs in cases that require them, and those who are opposed to "fragmenting the language" by recognizing that implementations intended for some purposes should offer semantics that could not be practically supported by those intended for some other purposes. IMHO, the Standard would be most useful if the third group could be excluded, but a Standard which excludes one of the others, *and is recognized as making no effort to satisfy that groups' needs*, would be better than what we have now. – supercat Oct 04 '19 at 15:48
  • Thank you for your elaborations. I personally think that introducing restrictions on the language to facilitate optimizations is disastrous, completely against the spirit of C. Not to even mention that the SAR is basically inviting programmers to fall into carefully prepared traps...of the worst kind, which might exhibit behaviours that are traceable only if one understands a specific(!) compilers optimization. That's supposed to be portable? To be more specific: If I allocate some memory (malloc), I expect that memory to be MINE, to do with it as I see fit. It can be very useful... – GermanNerd Oct 04 '19 at 16:14
  • @GermanNerd: N1570 6.5p7 would be fine except for one word: "by". If that were replaced by "...with a visible relation to...", a footnote made clear that the ability to recognize relations was a Quality of Implementation issue, the rule would be fine. I share your feeling about vandals stealing the language. – supercat Oct 04 '19 at 16:31
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/200406/discussion-between-germannerd-and-supercat). – GermanNerd Oct 04 '19 at 16:38