2

I was looking at the GNU implementation of obstacks, and I noticed the obstack_free subroutine is using pointer comparison to the beginnings and ends of the previous links of the linked list to find what block the pointer-to-be freed belongs to.

https://code.woboq.org/userspace/glibc/malloc/obstack.c.html

while (lp != 0 && ((void *) lp >= obj || (void *) (lp)->limit < obj))
{
     plp = lp->prev;
     CALL_FREEFUN (h, lp);
     lp = plp;
     h->maybe_empty_object = 1;
} //...

Such comparison appears to be undefined as per http://port70.net/~nsz/c/c11/n1570.html#6.5.8p5:

When two pointers are compared, the result depends on the relative locations in the address space of the objects pointed to. If two pointers to object types both point to the same object, or both point one past the last element of the same array object, they compare equal. If the objects pointed to are members of the same aggregate object, pointers to structure members declared later compare greater than pointers to members declared earlier in the structure, and pointers to array elements with larger subscript values compare greater than pointers to elements of the same array with lower subscript values. All pointers to members of the same union object compare equal. If the expression P points to an element of an array object and the expression Q points to the last element of the same array object, the pointer expression Q+1 compares greater than P. In all other cases, the behavior is undefined.

Is there a fully standard compliant way to implement obstacks. If not, what platforms could such comparison practically break on?

Petr Skocik
  • 58,047
  • 6
  • 95
  • 142
  • `memcmp` seems to be the only way in the standard. By which I mean it won't be outright UB. But it may not give a total order. – StoryTeller - Unslander Monica Dec 16 '17 at 10:53
  • 1
    You should link the source code and copy an excerpt from it into the question! – Antti Haapala -- Слава Україні Dec 16 '17 at 10:56
  • 1
    The platform where it may not work is the 16-bit x86, which uses segmented addressing. – user3386109 Dec 16 '17 at 10:59
  • @AnttiHaapala Done. – Petr Skocik Dec 16 '17 at 11:02
  • the problem with memcmp is little- and big-endian addressing - it doesn't give total ordering on a little-endian machine... which most are by now... I'd say: convert to `uintptr_t` and hope it works. – Antti Haapala -- Слава Україні Dec 16 '17 at 11:03
  • @user3386109 So practically I can expect this to work on all platforms that don't do segmented addressing, IOW basically all modern platforms? – Petr Skocik Dec 16 '17 at 11:03
  • 1
    No, you cannot absolutely expect this to work, because it is UB. The problem is that a compiler for platform can implement for example range checking. – Antti Haapala -- Слава Україні Dec 16 '17 at 11:04
  • 1
    Antti is correct that it's UB, and therefore not technically portable. However the specification has this to say about UB: *"Possible undefined behavior ranges from ignoring the situation completely with unpredictable results, **to behaving** during translation or program execution **in a documented manner characteristic of the environment** (with or without the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message)."* So, yes, practically it will work on platforms that use flat addressing, which is most modern platforms. – user3386109 Dec 16 '17 at 11:17
  • Thanks. If you put that in an answer and include that quote from the standard, I'll +1 and accept it. – Petr Skocik Dec 16 '17 at 11:24
  • Thanks for the offer, but I'll defer for now. Let's see what other people come up with, if anything. I'll check back tomorrow. – user3386109 Dec 16 '17 at 11:30
  • 1
    @user3386109: “A documented manner characteristic of the environment” does not mean a manner documented by the hardware the C implementation is running on. It means a manner documented by the C implementation. If you do not have a statement from the C implementation saying something about comparing pointers, you do not have the documentation this statement means. – Eric Postpischil Dec 16 '17 at 11:38
  • @user3386109 that's not what I said! Eric is right here. The platform doesn't specify the beliefs of the compiler. x86 allows addressing array out of bounds, but a compiler can generate code for range checks (and will) – Antti Haapala -- Слава Україні Dec 16 '17 at 12:13
  • If I write a simple compiler, I am very likely to use the machine properties directly. And it is easy for a compiler user to assume that, because the machine the programs run on has a flat hardware address space, that the compiler uses that address space for its pointers. But compiler developers now have decades of experience, and today’s compilers are no longer simple compilers. They have advanced features, they manipulate programs with mathematical abstractions, and they are built with multiple layers of complex software. Advanced features, such as bounds checking, may mean pointers are… – Eric Postpischil Dec 16 '17 at 13:13
  • … implemented with more than simple hardware addresses. And optimization occurs on a level far above the hardware instructions, producing transformations in the code that are not obvious to human eyes. Each transformation built into the optimizer of a compiler is valid according to the rules of C but can break a program that does not obey the rules of C. If a program compares two pointers from different objects, the optimizer is allowed to transform that code to any other code. – Eric Postpischil Dec 16 '17 at 13:16
  • I actually think pointer comparisons are one of the safer ways to violate the rules of C, because compiler writers know that certain software (software for the kernel, software for support libraries that must implement various memory management, and so on) needs to work with hardware addresses. So they may design the compiler to respect these uses of pointers even though the C standard does not define them. Nonetheless, reasoning that because the hardware behaves a certain way, C necessarily follows is logically incorrect. It is better to have a statement from the C implementation. – Eric Postpischil Dec 16 '17 at 13:18
  • As a general rule, it is unwise to take C that was written more than 20 years ago as a guide for what is and is not safe / good style / well-defined / ... to do today. – zwol Dec 16 '17 at 15:10
  • @EricPostpischil Yes, the last sentence of my previous comment should have been more nuanced, but I had reached the character limit, and I wasn't attempting to write an essay in the comments. – user3386109 Dec 16 '17 at 17:54
  • Would it be possible to use the "compare-not-equals-to-one-byte-at-a-time" trick (as [used here](https://stackoverflow.com/questions/4023320)) to avoid the less-than/greater-then comparisons (and hence the UB)? It would certainly be the most inefficient solution, but since you only asked for existence... – anol Dec 16 '17 at 21:50
  • glibc only aims to be compilable by GNU C ; it does not aim for ISO C conformance. (In fact that would be impossible since in ISO C you can not define functions with the same name as standard library functions). In other words, the standard applies to *implementations* which means a compiler that already has a standard library. Building the standard library is not covered by the standard – M.M Dec 17 '17 at 20:47

1 Answers1

1

I am not a language-lawyer, so I don't know how to answer OP's question, except that a plain reading of the standard does not describe the entire picture.

While the standard says that comparing unrelated pointers yields undefined results, the behaviour of a standards-compliant C compiler is much more restricted.

The first sentence in the section concerning pointer comparison is

When two pointers are compared, the result depends on the relative locations in the address space of the objects pointed to.

and for a very good reason.

If we examine the possibilities how the pointer comparison code may be used, we find that unless the compiler can determine which objects the compared pointers belong to at compile time, all pointers in the same address space must compare arithmetically, according to the addresses they refer to.

(If we prove that a standards-compliant C compiler is required by the standard to provide specific results when a plain reading of the C standard itself says the results are undefined, is such code standards-compliant or not? I don't know. I only know such code works in practice.)


A literal interpretation of the standard may lead to one believing that there is absolutely no way of determining whether a pointer refers to an array element or not. In particular, observing

int is_within(const char *arr, const size_t len, const char *ptr)
{
    return (ptr >= arr) && (ptr < (arr + len));
}

a standards compliant C compiler could decide that because comparison between unrelated pointers is undefined, it is justified in optimizing the above function into

int is_within(const char *arr, const size_t len, const char *ptr)
{
    if (size)
        return ptr != (arr + len);
    else
        return 0;
}

which returns 1 for pointers within array const char arr[len], and zero at the element just past the end of the array, just like the standard requires; and 1 for all undefined cases.

The problem in that line of thinking arises when a caller, in a separate compilation unit, does e.g.

char  buffer[1024];
char *p = buffer + 768;
if (is_within(buffer, (sizeof buffer) / 2, p)) {
    /* bug */
} else {
    /* correct */
}

Obviously, if the is_within() function was declared static (or static inline), the compiler could examine all call chains that end up in is_within(), and produce correct code.

However, when is_within() is in a separate compilation unit compared to its callers, the compiler can no longer make such assumptions: it simply does not, and cannot know, the object boundaries beforehand. Instead, the only way it can be implemented by a standards-compliant C compiler, is to rely on the addresses the pointers refer to, blindly; something like

int is_within(const char *arr, const size_t len, const char *ptr)
{
    const uintptr_t  start = POINTER_TO_UINTPTR(arr);
    const uintptr_t  limit = POINTER_TO_UINTPTR(arr + len);
    const uintptr_t  thing = POINTER_TO_UINTPTR(ptr);

    return (thing >= start) && (thing < limit);
}

where the POINTER_TO_UINTPTR() would be a compiler-internal macro or function, that converts the pointer losslessly to an unsigned integer value (with the intent that there would be a corresponding UINTPTR_TO_POINTER() that could recover the exact same pointer from the unsigned integer value), without consideration for any optimizations or rules allowed by the C standard.


So, if we assume that the code is compiled in a separate compilation unit to its users, the compiler is forced to generate code that provides more quarantees than a simple reading of the C standard would indicate.

In particular, if arr and ptr are in the same address space, the C compiler must generate code that compares the addresses the pointers point to, even if the C standard says that comparison of unrelated pointers yields undefined results; simply because it is at least theoretically possible for an array of objects to occupy any subregion of the address space. The compiler just cannot make assumptions that break conforming C code later on.

In the GNU obstack implementation, the obstacks all exist in the same address space (because of how they are obtained from the OS/kernel). The code assumes that the pointers supplied to it refer to these objects. Although the code does return an error if it detects that a pointer is invalid, it does not guarantee it always detects invalid pointers; thus, we can ignore the invalid pointer cases, and simply assume that because all obstacks are from the same address space, so are all the user-supplied pointers.


There are many architectures with multiple address spaces. x86 with a segmented memory model is one of these. Many microcontrollers have Harvard architecture, with separate address spaces for code and data. Some microcontrollers have a separate address space (different machine instructions) for accessing RAM and flash memory (but capable of executing from both), and so on.

It is even possible for there to be an architecture where each pointer has not only its memory address, but some kind of unique object ID associated with it. This is nothing special; it just means that on such an architecture, each object has their own address space.

Nominal Animal
  • 38,216
  • 5
  • 59
  • 86