va_arg returns different values on x86 and ARM

Question

I'm a new developer on a team who was just given a new Macbook with an M1 Pro (ARM). The rest of my team uses Intel Macbooks (x86). I ran a test which was supposed to create a file and write to it, however, the test errored out because the file was created with incorrect permissions.

I traced it back to a wrapper function that the team uses around open, sysio::openc. It makes use of a variadic template, and intends to eventually call open in libc. Normally, _syscall(fst, args...) would be used to call libc's open. However, for reasons outside the scope of this question, there is another open in available to our test, defined in our_open.cpp below, which is called by _syscall(fst, args...). That might make things a bit confusing, but I think it provides valuable debug information.

We've narrowed down the problem to the following: When debugging on my coworker's Intel (x86) Mac, inside our version of open, mode = va_arg(ap, int) returns 438 as expected. On my M1 Pro (ARM) Mac, mode = va_arg(ap, int) always returns 32.

Note

• The same permissions issue occurs if _syscall(fst, args...) directly calls libc's version of open instead of our redefined version. I left our redefined version of open in the question because we thought it adds debugging value to be able to look into where va_arg is called.
• Replacing our variadic sysio::openc wrapper with a direct call to libc's open fixes the issue and creates a file with the expected permissions.

It could have something to do with undefined behavior due to an unexpected type being passed to va_arg like from Why va_arg() produce different effects on x86_64 and arm?. However, I've experimented with things like mode_t mode = static_cast<mode_t>(va_arg(ap, int)) with no success. Also, it wouldn't explain why calling libc's open directly from _syscall(fst, args...) (avoiding our use of va_arg) still doesn't work.

<test.cpp>

const int fdOut = sysio::openc(path, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL | O_CLOEXEC, 438);

<wrapper.h>

template<IOType IOType, typename RetVal, typename... Args>
class IoWrapper
{
public:
    explicit IoWrapper(RetVal syscall(Args...))
      : _syscall(syscall)
    {}

    template<typename Arg1, typename... Ts>
    RetVal operator()(Arg1 fst, Ts... args)
    {
        const RetVal ret = _syscall(fst, args...);
        if (ret == RetVal(-1)) {
            abortOnVfsError<IOType>(fst, errno);
        }

        return ret;
    }

private:
    std::function<RetVal(Args...)> _syscall;
};
template<IOType IOType, typename RetVal, typename... Args>
IoWrapper<IOType, RetVal, Args...> make_wrapper(RetVal syscall(Args...))
{
    return IoWrapper<IOType, RetVal, Args...>(syscall);
}

inline auto openc = make_wrapper<IOType::read>((int (*)(const char*, int, int))::open);

<our_open.cpp>

extern "C" int open(const char* file, int flags, ...)
{
    static libc_open_ptr libc_open = (libc_open_ptr)dlsym(RTLD_NEXT, "open");
    int mode = 0;
    if ((flags & O_CREAT)) {
        va_list ap;
        va_start(ap, flags);
        mode = va_arg(ap, int); // Returns 438 (correct) on x86 but 32 on ARM (M1 Pro)
        va_end(ap);
    }

    return libc_open(file, flags, mode);
}

Answer 1

It is undefined behavior to call a function through a function pointer of the wrong type. In this case, open has signature int(const char*, int, ...) and is being called through a int (*)(const char*, int, int). I would say it's a miracle this code ever works on any architecture! The (IMO quite confusing, consider not using the ::) syntax (int (*)(const char*, int, int))::open is a C-style cast to int (*)(const char*, int, int) applied to ::open. C-style casts are dangerous when used with pointers and references because they suppress warnings about questionable casts. A static_cast would have errored here and a reinterpret_cast would have at least made you question your choices.

The simplest solution is probably to rely on std::function to do the adapting.

template<IOType IOType, typename RetVal, typename... Args>
class IoWrapper {
    std::function<RetVal(Args...)> syscall;
public:
    explicit IoWrapper(std::function<RetVal(Args...)> syscall) // function pointers are annoying anyway...
    : syscall(std::move(syscall))
    {}

    template<typename Arg1, typename... Ts>
    RetVal operator()(Arg1 fst, Ts... args) {
        const RetVal ret = syscall(fst, args...);
        if (ret == RetVal(-1)) {
            abortOnVfsError<IOType>(fst, errno);
        }
        return ret;
    }
};
// make_wrapper is unnecessary here because the argument types can't be inferred

inline IoWrapper<IOType::read, int, const char*, int, int> openc(open);

Note that BOTH your version of open and the standard one in the C library should have the signature int open(const char*, int, ...), hence they should either both work or both not work.