2

I'm knee deep in a project to connect Windows OpenVR applications running in Wine directly to Linux native SteamVR via a Winelib wrapper, the idea being to sidestep all the problems with trying to make what is effectively a very complicated device driver run inside Wine itself. I pretty much immediately hit a wall. The problem appears to be related to calling conventions, though I've had trouble getting meaningful information out of winedbg, so there's a chance I'm way way off.

The OpenVR API is C++ and consists primarily of classes filled with virtual methods. The application calls a getter (VR_GetGenericInterface) to acquire a derivative class object implementing those methods from the (closed source) runtime. It then calls those methods directly on the object given to it.

My current attempt goes like this: My wrapped VR_GetGenericInterface returns a custom wrapper class for the requested interface. This class's methods are defined in their own file separately compiled with -mabi=ms. It calls non-member methods in a separate file that is compiled without -mabi=ms, which finally make the corresponding call into the actual runtime.

This seems to work, until the application calls a method that returns a struct of some description. Then the application segfaults on the line the call happened, apparently just after the call returned, as my code isn't anywhere on the stack at that point and I verified with printfs that my wrapped class reaches the point of returning to the app.

This leads me to think there's yet another calling convention gotcha I haven't accounted for yet, related to structs and similar complex types. From googling it appears returning a struct is one of the more poorly-defined things in ABIs typically, but I found no concrete answers or descriptions of the differences.

What is most likely happening, and how can I dig deeper to see what exactly the application is expecting?

roothorick
  • 21
  • 1
  • You have to use the same ABI for any components that you want to link. Application binary interfaces are literally what an ABI is for. – Ben Jun 09 '17 at 07:44
  • I don't have that luxury. This project by definition needs to link together two components with incompatible ABIs. – roothorick Jun 09 '17 at 16:00
  • OK it looks like the `-mabi` flag sets the default abi, but you can flag each function individually also: https://gcc.gnu.org/onlinedocs/gcc/x86-Function-Attributes.html – Ben Jun 09 '17 at 16:37
  • That's what I've been doing, but it appears that `-mabi=ms` and `__attribute__((sysv_abi))` decorations aren't enough. It works until the application expects a struct out of the API, at which point I get a segfault, which appears to happen between when the stack is unwound to return to the application and when control should have been returned to the application. – roothorick Jun 09 '17 at 17:34
  • Another thing I had forgotten about: in an attempt to narrow it down, I made the wrapped class, instead of calling the non-member method to get into the API, simply return the struct with dummy values. That also crashes in the same way. So it appears that the return from my code back into the app is where the disconnect is. – roothorick Jun 09 '17 at 17:37
  • Are you also decorating the function with `stdcall`? – Ben Jun 10 '17 at 11:42

0 Answers0