How to fix "Access Violation" error when returning PyObject from C function using PyTCC library

Question

I am using a Python library that wraps LibTCC called PyTCC.

I am experimenting with ways to JIT compile code in Python. Problem is, when calling a function I can return normal C data types correctly but I get an "Access Violation" error when returning any PyObject *.

I have made sure that code can execute from PyTCC as my code example shows. This also means that the code example is compiling successfully.

import ctypes, pytcc

program = b"""
#include "Python.h"

/* Cannot return 3 due to access violation */
PyObject * pop(PyObject * self, PyObject * args, PyObject * kwargs) {
    // Cannot return *any* Python object
    return PyLong_FromLong(3);
}

int foobar() { return 3; }  // Returns 3 just fine

// Needed to appease TCC:
int main() { }
"""

jit_code = pytcc.TCCState()
jit_code.add_include_path('C:/Python37/include')
jit_code.add_library_path('C:/Python37')
jit_code.add_library('python37')
jit_code.compile_string(program)
jit_code.relocate()

foobar_proto = ctypes.CFUNCTYPE(ctypes.c_int)
foobar = foobar_proto(jit_code.get_symbol('foobar'))

print(f'It works: {foobar()}')

pop_proto = ctypes.CFUNCTYPE(ctypes.c_voidp)
pop = pop_proto(jit_code.get_symbol('pop'))

print('But this does not for some reason:')
print(pop())
print('Never gets here due to access violation :(')

The output of the program should be:

It works: 3
But this does not for some reason:
3
Never gets here due to access violation :(

But instead, I am getting this exact error:

It works: 3
But this does not for some reason:
Traceback (most recent call last):
  File "fails.py", line 40, in <module>
    print(pop())
OSError: exception: access violation writing 0x00000000FFC000E9

Answer 1

Most likely it's because you don't have the GIL when creating the object. You also have an issue with the return type. ctypes.c_voidp tells python to treat it like an int instead of a PyObject, so all you'd see if it wasn't for the access violation is the value pointer itself not what it's pointing at.

Try:

    PyObject * pop() {
    PyGILState_STATE gstate;
    gstate = PyGILState_Ensure();
    PyObject* obj = PyLong_FromLong(10);
    PyGILState_Release(gstate);
    return obj;
}

and switch
pop_proto = ctypes.CFUNCTYPE(ctypes.c_voidp)
to
pop_proto = ctypes.CFUNCTYPE(ctypes.py_object)

output from my run (changed the value from 3 to 10 in the pyobject just to show it made it)

It works: 3
But this does not for some reason:
10
Never gets here due to access violation :(

Answer 2

Didn't work with PyTCC, but there's something wrong in the code.

According to [Python 3]: class ctypes.PyDLL(name, mode=DEFAULT_MODE, handle=None) (emphasis is mine):

Instances of this class behave like CDLL instances, except that the Python GIL is not released during the function call, and after the function execution the Python error flag is checked. If the error flag is set, a Python exception is raised.

Thus, this is only useful to call Python C api functions directly.

Note: CFUNCTYPE is for CDLL, same thing that PYFUNCTYPE is for PyDLL.

As a consequence, in pop_proto, you should replace ctypes.CFUNCTYPE with ctypes.PyFUNCTYPE (note that you have a typo in c_voidp).

Next, same page states that for PyObject* (C), py_object should be used (Python). So:

pop_proto = ctypes.PyFUNCTYPE(ctypes.py_object)

If you want to be rigorous, you'll have to include the arguments in the prototype, which would make the code look a bit more complicated, but for this particular case (they are ignored), it's not mandatory:

pop_proto = ctypes.PyFUNCTYPE(ctypes.py_object, ctypes.py_object, ctypes.py_object, ctypes.py_object)

Here's an example for PyObject *PyBytes_Repr(PyObject *obj, int smartquotes) (calling the C function in the "old fashioned" way):

[cfati@CFATI-5510-0:C:\WINDOWS\system32]> "e:\Work\Dev\VEnvs\py_064_03.07.03_test0\Scripts\python.exe"
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import sys
>>> import os
>>> import ctypes
>>>
>>> python_dll_name = os.path.join(os.path.dirname(sys.executable), "python" + str(sys.version_info.major) + str(sys.version_info.minor) + ".dll")
>>> python_dll_name
'e:\\Work\\Dev\\VEnvs\\py_064_03.07.03_test0\\Scripts\\python37.dll'
>>>
>>> python_dll = ctypes.PyDLL(python_dll_name)
>>>
>>> pybytes_repr_proto = ctypes.PYFUNCTYPE(ctypes.py_object, ctypes.py_object, ctypes.c_int)
>>> pybytes_repr = pybytes_repr_proto(("PyBytes_Repr", python_dll))
>>>
>>> b = b"abcd"
>>>
>>> reprb = pybytes_repr(b, 0)
>>> reprb
"b'abcd'"

You might also check [SO]: How to cast a ctypes pointer to an instance of a Python class (@CristiFati's answer).