Undefined symbols when trying to use Cython Extensions

Question

I am sure that there is something simple I am missing, but I have been staring at it long enough that I'm not going to see it. I have read the questions and answers here, here, here, here, and here without encountering a solution for my use case. I am trying to put together a simple test structure for playing with smart pointers and cython (although I can't even get to the smart pointer part). Code is as follows:

test_unique_ptr.h

struct TestStruct {
    int a;
    int b;
};

class TestClass
{
public:
    TestClass(void);
    ~TestClass(void);

    TestStruct myts;

    int getA() { return myts.a; }
    int getB() { return myts.b; }
};

test_unique_ptr.cpp

#include "test_unique_ptr.h"

TestClass::TestClass(void) {
    myts.a = 4;
    myts.b = 7;
}

This compiles successfully into libTestUPtr.so

test_u_ptr.pyx

from libcpp.memory cimport unique_ptr
from cython.operator cimport dereference as deref

cdef extern from "test_unique_ptr.h":
    ctypedef struct TestStruct:
        int a
        int b

    cdef cppclass TestClass:
        TestClass()
        TestStruct foo
        int getA()
        int getB()

cdef class TestClass1:
    cdef:
        TestClass tc

    def __cinit__(self):
        self.tc = TestClass()

    def getValue(self):
        print(self.tc.getA())

foo = TestClass1()
foo.getValue()

setup.py

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy

setup(
    cmdclass = {'build_ext': build_ext},
    ext_modules = [
        Extension("test_u_ptr",
            language="c++",
            sources=["test_u_ptr.pyx"],
            include_dirs=["/home/share/data/code/python/test/include", numpy.get_include()],
            libraries = ["TestUPtr"],
            library_dirs=["/home/share/data/code/python/test/lib"],
            extra_compile_args=['-lstdc++','-std=c++11', '-v'],
            extra_link_args=['-lstdc++', '-v'],
            )
        ]
    )

This also compiles successfully into test_u_ptr.cpython-35m-x86_64-linux-gnu.so. Now comes the fun part.

$ python3
>>> import test_u_ptr
ImportError: /home/share/data/code/python/test/test_u_ptr.cpython-35m-x86_64-linux-gnu.so: undefined symbol: _ZN9TestClassD1Ev

The symbols appear to be properly defined in libTestUPtr.so

$ nm -C libTestUPtr.so

0000000000201030 B __bss_start

0000000000201030 b completed.6973
                 w __cxa_finalize
0000000000000580 t deregister_tm_clones
00000000000005f0 t __do_global_dtors_aux
0000000000200e38 t __do_global_dtors_aux_fini_array_entry
0000000000201028 d __dso_handle
0000000000200e48 d _DYNAMIC
0000000000201030 D _edata
0000000000201038 B _end
0000000000000680 T _fini
0000000000000630 t frame_dummy
0000000000200e30 t __frame_dummy_init_array_entry
0000000000000700 r __FRAME_END__
0000000000201000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
0000000000000530 T _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000200e40 d __JCR_END__
0000000000200e40 d __JCR_LIST__
                 w _Jv_RegisterClasses
00000000000005b0 t register_tm_clones
0000000000201030 d __TMC_END__
0000000000000670 T TestClass::TestClass()
0000000000000670 T TestClass::TestClass()

However, almost everything is undefined in the cython library.

$ nm -C test_u_ptr.cpython-35m-x86_64-linux-gnu.so

0000000000204530 B __bss_start
0000000000204540 b completed.6973
                 w __cxa_finalize@@GLIBC_2.2.5
0000000000001be0 t deregister_tm_clones
0000000000001c50 t __do_global_dtors_aux
0000000000203d40 t __do_global_dtors_aux_fini_array_entry
0000000000204180 d __dso_handle
0000000000203d50 d _DYNAMIC
0000000000204530 D _edata
00000000002046f8 B _end
0000000000003154 T _fini
0000000000001c90 t frame_dummy
0000000000203d38 t __frame_dummy_init_array_entry
00000000000035b8 r __FRAME_END__
0000000000204000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
00000000000015e8 T _init
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
0000000000203d48 d __JCR_END__
0000000000203d48 d __JCR_LIST__
                 w _Jv_RegisterClasses
                 U PyBaseObject_Type
                 U PyBytes_FromStringAndSize
                 U PyCapsule_New
                 U PyCFunction_Type
                 U _Py_CheckRecursionLimit
                 U _Py_CheckRecursiveCall
                 U PyCode_New
                 U PyDict_GetItem
                 U PyDict_GetItemString
                 U PyDict_New
                 U PyDict_SetItem
                 U PyDict_SetItemString
                 U PyErr_Format
                 U PyErr_Occurred
                 U PyErr_SetString
                 U PyErr_WarnEx
                 U PyEval_EvalCodeEx
                 U PyEval_EvalFrameEx
                 U PyExc_ImportError
                 U PyExc_NameError
                 U PyExc_SystemError
                 U PyExc_TypeError
                 U PyFrame_New
                 U PyFunction_Type
                 U Py_GetVersion
                 U PyImport_AddModule
                 U PyImport_GetModuleDict
00000000000023d0 T PyInit_test_u_ptr
                 U PyLong_FromLong
                 U PyMem_Malloc
                 U PyMem_Realloc
                 U PyMethod_Type
                 U PyModule_Create2
                 U PyModule_GetDict
                 U _Py_NoneStruct
                 U PyObject_Call
                 U PyObject_CallFinalizerFromDealloc
                 U PyObject_GetAttr
                 U PyObject_SetAttrString
                 U PyOS_snprintf
                 U PyThreadState_Get
                 U PyTraceBack_Here
                 U PyTuple_New
                 U PyTuple_Pack
                 U PyType_Ready
                 U PyUnicode_Decode
                 U PyUnicode_FromFormat
                 U PyUnicode_FromString
                 U PyUnicode_FromStringAndSize
                 U PyUnicode_InternFromString
0000000000204560 B __pyx_module_is_main_test_u_ptr
0000000000001c10 t register_tm_clones
                 U __stack_chk_fail@@GLIBC_2.4
0000000000204530 d __TMC_END__
000000000000333b r __pyx_k_end
0000000000003337 r __pyx_k_foo
0000000000204650 b __pyx_print
0000000000003332 r __pyx_k_file
0000000000003329 r __pyx_k_main
0000000000003320 r __pyx_k_test
00000000002046a0 b __pyx_lineno
0000000000204690 b __pyx_clineno
000000000000331a r __pyx_k_print
0000000000204580 b __pyx_methods
0000000000204630 b __pyx_n_s_end
0000000000204620 b __pyx_n_s_foo
0000000000204680 b __pyx_filename
0000000000204628 b __pyx_n_s_file
0000000000204600 b __pyx_n_s_main
00000000002045d0 b __pyx_n_s_test
0000000000204320 d __pyx_moduledef
00000000002045f0 b __pyx_n_s_print
0000000000204660 b __pyx_code_cache
0000000000003311 r __pyx_k_getValue
00000000002041a0 d __pyx_string_tab
00000000002046b0 b __pyx_empty_bytes
00000000002046c0 b __pyx_empty_tuple
0000000000001e10 t __Pyx_AddTraceback(char const*, int, int, char const*)
0000000000003302 r __pyx_k_pyx_vtable
0000000000204610 b __pyx_n_s_getValue
00000000002046a8 b __pyx_empty_unicode
00000000000018e0 t __Pyx_PyObject_Call(_object*, _object*, _object*)     
00000000002045e0 b __pyx_n_s_pyx_vtable
00000000000019c1 t __Pyx_PyFunction_FastCallDict(_object*, _object**, int,     _object*) [clone .constprop.3]
00000000002043a0 d __pyx_type_10test_u_ptr_TestClass1
0000000000204640 b __pyx_ptype_10test_u_ptr_TestClass1
0000000000001d30 t __pyx_tp_new_10test_u_ptr_TestClass1(_typeobject*, _object*, _object*)
00000000002045c0 b __pyx_vtable_10test_u_ptr_TestClass1
00000000002045a0 b __pyx_methods_10test_u_ptr_TestClass1
0000000000204670 b __pyx_vtabptr_10test_u_ptr_TestClass1
0000000000001ce0 t __pyx_tp_dealloc_10test_u_ptr_TestClass1(_object*)
0000000000002230 t __pyx_f_10test_u_ptr_10TestClass1_getValue(__pyx_obj_10test_u_ptr_TestClass1*)
0000000000001cd0 t __pyx_f_10test_u_ptr_10TestClass1_printValue(__pyx_obj_10test_u_ptr_TestClass1*)
00000000002046d0 b __pyx_b
00000000002046e0 b __pyx_d
00000000002046f0 b __pyx_m
                 U TestClass::TestClass()
                 U TestClass::~TestClass()

Both libraries are 64 bit.

$ file libTestUPtr.so 
libTestUPtr.so: ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=4c6eb8560a244ebd485d483f1c27004145ad1882, not stripped

$ file test_u_ptr.cpython-35m-x86_64-linux-gnu.so 
test_u_ptr.cpython-35m-x86_64-linux-gnu.so: ELF 64-bit LSB  shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=5a2daa515d4a22dd46484968e817cd3a3bcebe5a, not stripped

libTestUPtr.so is in LD_LIBRARY_PATH and at the correct location

$ echo $LD_LIBRARY_PATH 
/home/share/data/code/python/test/lib

$ ls /home/share/data/code/python/test/lib/
libTestUPtr.so

I am at a loss as to what is going on. Any insight would be greatly appreciated.

gcc is version 4.8.4. Python is version 3.5.3. Cython is version 0.25.2.

Adding the linker option "-Wl,--no-undefined" appears to have fixed the problem. I have absolutely no idea why. I'm going to leave this open for the time being in case anyone can shed some light on what the heck is going on. — PaxRomana99, Aug 13 '17 at 00:45
I think it should be `def getValue()` and not `cdef getValue()` in your pyx-file. — ead, Aug 13 '17 at 09:12

ead · Accepted Answer · 2017-08-13T09:33:17.033

There is nothing mystical about your problem and I doubt, that adding --no-undefined has fixed it.

The missed symbol is the destructor of the class TestClass. Your shared library provides two symbols for the TestClass: But both these symbols are constructors (I don't fully understand why there are two and why both symbols have the same address) and none of them is the needed destructor.

The origin of this problem is that you declare your destructor, so it is no longer default-implemented by the compiler, but you don't provide an implementation in the cpp-file.

So your options are:

delete the declaration of the destructor from the h-file, let the compiler take care of it.
implement your destructor in the cpp-file.

Which one is better depends on context.

That gcc emits two symbols for the constructor and two for the destructor seems to be a known missed optimization, here is a great explanation.

The addresses are the same, because complier aliases both identical methods:

_ZN9TestClassC2Ev: //Constructor of the BaseObject-constructor
.LFB3:
    ...
    ret
.LFE3:
    ..
    .globl  _ZN9TestClassC1Ev
    .set    _ZN9TestClassC1Ev,_ZN9TestClassC2Ev //HERE WE GO, Complete Object constructor which we use normally

You are correct. The problem was the missing destructor and I don't think the --no-undefined actually had anything to do with fixing it. — PaxRomana99, Aug 13 '17 at 17:03

Undefined symbols when trying to use Cython Extensions

1 Answers1