This is not valid code, the garbage collector can only update the value of the vrt_ptr
and tris_ptr
variables. But the unmanaged code uses a copy of these pointers, the value of the copy cannot be updated by the GC. So if a garbage collection occurs while the unmanaged code is running, possible for example when other threads in the program trigger a collection, then the unmanaged code will read garbage data through the pointer copy. Very hard to diagnose, it doesn't happen very often.
You must pin the vertices and tris arrays. In your case already ably done by the pinvoke marshaller, simply by passing the arrays directly without using fixed. Fix:
double* ptr = compute(vertices, 5, tris, 5);
Adjust the pinvoke declaration accordingly, replacing double* with double[].
You'll now also have to the deal with the likely reason you wrote this code in the first place. There is no scenario where casting an int[] to double[] is ever valid, the likely reason you got a garbage result early before that GC disaster could strike. If you can't update the declaration of tris for some reason then you must create a double[] before the call.