1

In a C++/CLI assembly, I'm trying to call a managed delegate from a native callback. I followed Doc Brown's answer here, and my implementation so far looks like this:

The native callback - ignore the commented out parts for now:

static ssize_t idaapi idb_callback(void* user_data, int notification_code, va_list va)
{
    switch (notification_code)
    {
        case idb_event::byte_patched:
        {
            //ea_t address = va_arg(va, ea_t);
            //uint32 old_value = va_arg(va, uint32);
            return IdaEvents::BytePatched(0, 0);
        }
        break;
    }
    return 0;
}

As you can see above, I call this managed delegate instantiated in a static class:

public delegate int DatabaseBytePatchedHandler(int address, int originalValue);

private ref class IdaEvents
{
    static IdaEvents()
    {
        BytePatched = gcnew DatabaseBytePatchedHandler(&OnDatabaseBytePatched);
    }

    public: static DatabaseBytePatchedHandler^ BytePatched;

    private: static int OnDatabaseBytePatched(int address, int originalValue)
    {
        return 0;
    }
};

This compiles fine. But the code is incomplete - remember the commented out part in the native callback above? I actually have to retrieve the values from the va_list passed to the callback, and pass those on to my managed delegate:

            ea_t address = va_arg(va, ea_t);
            uint32 old_value = va_arg(va, uint32);
            return IdaEvents::BytePatched(address, old_value);

But as soon as I uncomment one of the lines using va_arg, I cannot compile the project anymore and retrieve the following errors marking the line where I call the managed delegate:

C3821   'IdaEvents': managed type or function cannot be used in an unmanaged function
C3821   'IdaEvents::BytePatched': managed type or function cannot be used in an unmanaged function
C3821   'BytePatched': managed type or function cannot be used in an unmanaged function
C3821   'DatabaseBytePatchedHandler::Invoke': managed type or function cannot be used in an unmanaged function
C3642   'int DatabaseBytePatchedHandler::Invoke(int,int)': cannot call a function with __clrcall calling convention from native code
C3175   'DatabaseBytePatchedHandler::Invoke': cannot call a method of a managed type from unmanaged function 'idb_callback'

This really confuses me. Why is the compiler suddenly acting up as soon as I try to use va_arg? Even a single line without any assignment causes this error to pop up.

Am I thinking too naive here? I'm obviously missing a piece of the puzzle, and any help supporting me in finding it is greatly appreciated.

Ray
  • 7,940
  • 7
  • 58
  • 90
  • Just a guess: if I got this [former SO post](https://stackoverflow.com/questions/17821794/how-do-i-pass-variable-arguments-from-managed-to-unmanaged-with-a-c-cli-wrappe) right, using variadic arguments is only possible for C functions, not for C++/CLI. Maybe you can solve this by addding another layer of indirection: an additional C callback which does not need va_list and va_arg? – Doc Brown Mar 15 '18 at 18:55
  • The core issue is that your "native callback" is not all that native. It is being compiled with /clr in effect and gets translated to MSIL and jitted at runtime. Which is why you could directly call a managed method. While that might sound convenient, it is not actually very desirable, you get the worst of both worlds. And as soon as you use va_arg then the show is over, that requires native code. Use a function pointer to go from native to managed: https://stackoverflow.com/a/2973278/17034 – Hans Passant Mar 15 '18 at 19:03
  • @HansPassant I was guessing I could not get around that (I've previously bookmarked your answer there but was stuck trying to implement a managed delegate accepting a `va_list` somehow, now knowing that's not possible), so I've experimented a bit: What do you think about calling a new "pseudo-native" stub `idb_byte_patched(ea_t address, uint32 old_value)` from the currently not-so-native callback above, which has "normal" parameters, and then call `IdaEvents::BytePatched` in there? I could compile and test that successfully. – Ray Mar 15 '18 at 21:12
  • Hmya, compiling native code to MSIL is nails on my black-board. They did a terrific job but left lots of land-mines scattered around by making it look easy. Do what works and makes you happy. – Hans Passant Mar 15 '18 at 21:27

0 Answers0