Exception handling in cython

Question

I have two .pyx files - bar.pyx and baz.pyx. I want to combine them into a single .so file.

In baz.pyx I have a function baz that should do some checks and raise an exception if something goes wrong. In bar.pyx I want to call baz() and expect exception to be raised with traceback printed.

Unfortunately, whatever I try, I get some other runtime errors.

Extensions in setup.py

[
    Extension(
        'testlib.baz', ['src/testlib/baz.pyx'],
    ),
    Extension(
        'testlib.foo', ['src/testlib/bar.pyx', 'src/testlib/baz.c'],
    ),
]

How tested

import testlib.foo
testlib.foo.foo_baz()

Variant 1

# baz.pyx

cdef public baz():
    raise ValueError

# bar.pyx

cdef extern baz()

def foo_baz():
    baz()   # Segmentation fault

Variant 2

# baz.pyx

cdef public int baz() except -1:
    PyErr_SetNone(ValueError)
    return -1

# bar.pyx

cdef extern int baz() except -1

def foo_baz():
    baz()   # SystemError: <built-in function foo_baz> returned NULL without setting an error

I can return some value from baz and raise an exception in foo_baz depending on return value, but I want as minimum logic to be present in bar.pyx.

# baz.pyx

cdef public int baz():
    return -1

# bar.pyx

cdef extern int baz()

def foo_baz():
    if baz() == -1:
        raise ValueError    # OK

This'll be because you're never importing the `baz` module. Cython sets up global variables for exceptions during module import. Since this global variable is never initialized it all goes wrong. The solution is probably not to try to force multiple modules into a single .so file: it isn't designed for it and it's surprisingly complicated to do right. — DavidW, Oct 22 '20 at 10:49

score 3 · Answer 1 · answered Oct 22 '20 at 09:44

3

Your except -1 variant should raise an exception the normal way instead of trying to manually set an exception and return an error value:

cdef public int baz() except -1:
    raise ValueError

Cython will handle the exception setting and the return value for you.

answered Oct 22 '20 at 09:44

user2357112

260,549
28
431
505

If I change it exactly this way, I get `Exception clause not allowed for function returning Python object`. If I remove types, etc I end up with Variant 1 with segmentation fault. – Tzoiker Oct 22 '20 at 09:59
@Tzoiker: That should only happen for functions that are actually declared to return Python objects, which `cdef public int ...` isn't. Are you sure you had the `int` there when you tested it? – user2357112 Oct 22 '20 at 10:02
My bad, sorry. Double-checked it, now compiles, but I get `segmentation fault python` as with Variant 1. Can it be connected with how I cythonize extensions? – Tzoiker Oct 22 '20 at 10:34

score 3 · Accepted Answer · answered Oct 22 '20 at 16:36

To expand my comment a bit more: Cython does quite a bit of work when the module is imported setting up C global variables to give it quite access to thinks like C builtin types (including exceptions), string constants, and a few other things. In this case the line is translated to

__Pyx_Raise(__pyx_builtin_ValueError, 0, 0, 0);

and if __pyx_builtin_ValueError isn't initialized then it all goes wrong.

The upshot is that Cython code (even public Cython code) isn't actually standalone but is part of a module and that does rely on the module init function being called for it to work. In your case it failed when raising an exception, but there's a range of similar bugs waiting to happen if you got past that.

As a general rule I'd advise against linking multiple modules together into one .so file. However it can be made to work; see https://stackoverflow.com/a/52714500/4657412 for a recipe. As you can see it's quite involved and requires some understanding of the C API import process.

Other options (that don't involve writing the code in C yourself) would be to use multiple modules and the cimport mechanism for sharing implementation between them, or possibly the older "include" mechanism which is usually not recommended but is sometimes convenient.

Exception handling in cython

Extensions in setup.py

How tested

Variant 1

Variant 2

2 Answers2