How to access Python dunder methods in C using CFFI?

Question

I was trying to follow this example to attempt to create a simple hash table in C that I could use in Python (using CFFI). Following are the corresponding files and relevant code.

hash_table.h:

typedef struct {
    size_t size;
    ...
    int cmp_function(const PyObject*, const PyObject*);
} hash_table_t;

build_hash_table.py:

from cffi import cffi

HEADER_FILE_NAME = "hash_table.h"
SOURCE_FILE_NAME = "hash_table.c"

ffi = FFI()

header = open(HEADER_FILE_NAME, "rt").read()
source = open(SOURCE_FILE_NAME, "rt").read()

ffi.set_source("hashtable.hash_table", header + source)
ffi.cdef(header)

if __name__ = "__main__":
    ffi.compile()

Now, everything so far works. However, I am going to want to insert whatever data types in the hash table. How do I get access to whatever object I am receiving within the C code? For example, say I had this class in Python:

class Person():

    def __init__(self, name, id):
        self.name = name
        self.id = id

    def __eq__(self, other):
        return self.id == other.id

When I am searching for Person objects in the hash table, how do I access the __eq__ method? As you can see above, I have a generic compare function declared as int cmp_function(const void*, const void*); in the hash table. The goal would be that every time I search for an object, I can know the __eq__ method definition of that object - in the C side.

Hope I made my question clear, thank you in advance!

Your objects should not be `void *` but `PyObject *` then... — Antti Haapala -- Слава Україні, Jul 05 '20 at 17:30
they should not be const. Anyway, the access requires CPython, doesn't work in PyPy etc... — Antti Haapala -- Слава Україні, Jul 05 '20 at 17:43
If I'm only using them for comparison, for what reason would I not want them to be const? Either way, I'm fine with the code only being compatible with CPython - of course, a more portable option would be good to know too :) — Saucy Goat, Jul 05 '20 at 17:46

score 1 · Accepted Answer · answered Jul 05 '20 at 18:45

1

See https://cffi.readthedocs.io/en/latest/using.html#extern-python-and-void-arguments. Avoid using PyObject * with CFFI. The correct answer is to keep int cmp_function(const void*, const void*);. I will assume that you already use ffi.new_handle() to convert the objects to the void * equivalent that you can send to C (in the add_to_hash_table() method, etc.) and ffi.from_handle() to read them back (in the get_from_hash_table() method, etc.).

To implement the comparison function, you declare a function in Python with extern "Python", as in the above document. Then you use that function for the cmp_function pointer in your struct. Inside that Python function, you receive the two arguments as two void *. You first convert them back to the original Python objects with ffi.from_handle(), and then just use regular Python---in your case, probably just if x == y: return 1; else: return 0 or similar.

Be careful about keeping the Python objects alive after calling ffi.new_handle(). They are not kept alive just by the fact that their void * representation happens to be stored inside some C structures.

Note that CFFI is not quite well suited to implement pure data structures, because all these conversions and multiple storages that increase the overhead when compared with the direct CPython-C-API approach.

answered Jul 05 '20 at 18:45

Armin Rigo

12,048
37
48

Sorry for the late response. I had not yet seen the `new_handle` and `from_handle` functions. By "declare a function in Python with `extern "Python"` you mean in the C header file? As to the comparison function, I need to know how to compare objects in the C side of things - if I'm inserting a new element in the hash table, I need to know if that element is already there. So I would need to "retrieve" the comparison function from Python. – Saucy Goat Jul 06 '20 at 16:01
Also regarding CFFI not being appropriate for the job, I followed [these docs](https://docs.python.org/3/extending/extending.html#:~:text=the%20Python%20Interpreter%20%C2%BB-,1.,library%20functions%20and%20system%20calls.), which said CFFI would increase portability. Thank you for the heads-up though :) – Saucy Goat Jul 06 '20 at 16:08
Quoting: "For example, if your use case is calling C library functions or system calls, you should consider using the ctypes module or the cffi library rather than writing custom C code." Note how it mentions invoking library or system calls, but not implementing a pure data structure. – Armin Rigo Jul 07 '20 at 20:45
About "declare a function in Python with `extern "Python"`: I mean as documented in detail at the URL https://cffi.readthedocs.io/en/latest/using.html#extern-python-and-void-arguments . – Armin Rigo Jul 07 '20 at 20:45

score 0 · Answer 2 · answered Jul 09 '20 at 17:07

0

For those who might be looking for another to compare Python objects in C, I found this question which explains exactly how to do that.

answered Jul 09 '20 at 17:07

Saucy Goat

1,587
1
11
32

How to access Python dunder methods in C using CFFI?

2 Answers2