Cython: undefined symbol in shared object

Question

I am importing c++ code in a python project, everything seems to compile just fine however when importing my .pyx I get:

from AGCython import *
ImportError: /path/to/shared/object/AGCython.cpython-36m-x86_64-linux-gnu.so: undefined symbol: c_Stat_GetMeanAndVariance_double

In my AGCython.pyx I have:

cdef extern void c_Stat_GetMeanAndVariance_double (double* array, int nSize, double* mean, double* var)

and its python wrapper

def Stat_GetMeanAndVariance_double(np.ndarray[double, ndim=1, mode="c"] input not None):
    cdef int m #nSize
    m = input.shape[0]
    cdef double mean, var
    c_Stat_GetMeanAndVariance_double(&input[0], m, &mean, &var)
    return mean, var

this cpp function is defined in AGc.cpp:

#include "AGc.h"
void c_Stat_GetMeanAndVariance_double(const double *aData, const int nSize, double &mean, double &var)
{
    // Special case, small vector
    if (nSize<=1)
    {
        var= 0;
        if (nSize)
            mean= *aData;
        else
            mean= 0;
        return;
    }

    double s, ssqr;
    Stat_GetSums_double(aData, nSize, s, ssqr);

    mean= s/nSize;
    var= Stat_GetVariance(s, ssqr, nSize);
    return;
}

and AGc.h contains:

void c_Stat_GetMeanAndVariance_double(const double *aData, const int nSize, double &mean, double &var);

I my compilation script is this:

from distutils.core import setup
from Cython.Build import cythonize
from distutils.extension import Extension
import numpy


sourcefiles = ['AGCython.pyx', 'AGc.cpp']

extensions = [Extension("AGCython", sourcefiles)]

setup(
    ext_modules = cythonize(extensions, annotate=True)
)

Which results in this gcc call:

gcc -pthread -B /home/ludvig/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ludvig/anaconda3/include/python3.6m -c AGc.cpp -o build/temp.linux-x86_64-3.6/AGc.o
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /home/ludvig/anaconda3/compiler_compat -L/home/ludvig/anaconda3/lib -Wl,-rpath=/home/ludvig/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/AGCython.o build/temp.linux-x86_64-3.6/AGc.o -o /path/to/my/project/AGCython.cpython-36m-x86_64-linux-gnu.so

I don't understand what I've missed here, I don't think the warning is the problem from reading this question

ead · Accepted Answer · 2018-05-17T19:40:14.607

3

Your problem is the name-mangling. There are however two reasons why it doesn't work.

First

C-language without name-mangling is used per default for AGCython.pyx, i.e. it expects the symbol to have the name c_Stat_GetMeanAndVariance_double.

The additional file is a *.cpp, thus gcc decides to compile it as C++-source-code, that means the name-mangling kicks in an the corresponding symbol name becomes _Z32c_Stat_GetMeanAndVariance_doublePKdiRdS1_. And thus looking for the non-mangled-name, the loader fails during the runtime.

There are different ways to fix it, but if you plan to use c++ anyway, the simplest would be to add language='c++' to your setup:

extensions = [Extension("AGCython", 
                        sourcefiles,
                        language='c++')]

Second:

You declare your exported function as:

cdef extern void c_Stat_GetMeanAndVariance_double (...)

It is translated from cython to

__PYX_EXTERN_C DL_IMPORT(void) c_Stat_GetMeanAndVariance_double(...);

And __PYX_EXTERN_C is a define for:

#ifndef __PYX_EXTERN_C
  #ifdef __cplusplus
    #define __PYX_EXTERN_C extern "C"
  #else
    #define __PYX_EXTERN_C extern
  #endif
#endif

That means it turns off the name-mangling for the C++-case. To avoid this you need to include the function from the header, as it is usually done:

cdef extern from "AGc.h":
    void c_Stat_GetMeanAndVariance_double(...)

edited May 17 '18 at 19:40

answered May 17 '18 at 10:15

ead

32,758
6
90
153

Hi! Thank you for the concise answer! I copied that into my compilation script but I still get the same problem though ... Is it possible to find the function in the compiled code and see if it has been mangled? – L.E. May 17 '18 at 10:52
@L.E. Rebuild everything - that normally doesn’t happen when setup script is changed – ead May 17 '18 at 10:55
I deleted the shared object, rebuilt but no luck. However; I noticed that in the .pyx I've defined mean and var in the extern function as pointer arguments whereas the c function wants mean and var as reference . I changed it to be consistent and get a different problem: `c_Stat_GetMeanAndVariance_double(&input[0], m, &mean, &var) ^ AGCython.pyx:497:58: Cannot assign type 'double *' to 'double'` I'm accepting your answer since it seems to be correct for the original question – L.E. May 17 '18 at 11:43
1

@L.E. deleting only *.so is not enough. You have also to rebuild pyx-file. This is done automatically only if the pyx-file has changed. – ead May 17 '18 at 11:53
Yes, I'm aware, it did rebuild; `running build_ext building 'AGCython' extension gcc -pthread -B /home/ludvig/anaconda3/compiler_compat` etc.... – L.E. May 17 '18 at 12:35

score 1 · Answer 2 · answered May 17 '18 at 13:52

1

What finally did the trick for me was changing

cdef extern void c_Stat_GetMeanAndVariance_double (double* array, int nSize, double &mean, double &var)

into:

cdef extern from "AGc.h":
    void c_Stat_GetMeanAndVariance_double(const double* aData, const int nSize, double &mean, double &var)

from looking at this page in the cython docs

Why this worked and not the other way I'm not sure about, however

answered May 17 '18 at 13:52

L.E.

45
6

1

Sorry, I somehow overlooked the way you declared your function, because usually it is donevia "extern from". Take a look at my update if you are interseted, why it didn't work in the original version. – ead May 17 '18 at 16:55
Excellent! Thanks for clarifying what happens 'under the hood' - much appreciated – L.E. May 17 '18 at 18:53

Cython: undefined symbol in shared object

2 Answers2