2

Intro

As a follow-up to my earlier question here, i still try to:

  • add new functionality to one of scipy's modules (e.g. scipy.optimize)
  • based on C++-code
    • which defines an interface-function to be called with ctypes
  • automatically setup through setup.py in scipy/optimize

Code incorporation

To keep it simple, let's assume we got the following (dumb and) simple C++-code:

extern "C" void hallo()
{
    int a = 0;
}

where we use the extern keyword to handle name-mangling.

We introduce new files and folders in scipy.optimize:

scipy/optimize/__init__py
scipy/optimize/lbfgsb/...
scipy/optimize/lbfgsb.py
...
scipy/optimize/_mylib/README.md       # new
scipy/optimize/_mylib/LICENSE         # new
scipy/optimize/_mylib/src/debug.cpp   # new
scipy/optimize/mylib.py               # new

Setup preparation

We prepare the module-setup in scipy/optimize/setup.py:

from __future__ import division, print_function, absolute_import

from os.path import join

from scipy._build_utils import numpy_nodepr_api

def configuration(parent_package='',top_path=None):
    from numpy.distutils.misc_util import Configuration
    from numpy.distutils.system_info import get_info
    config = Configuration('optimize',parent_package, top_path)

    # MODIFICATION START
    # OTHER EXTENSIONS OMITTED 
    # INSPIRED BY scipy.spatial: ckdtree/src & setup.py

    mylib_src = ['debug.cpp']
    mylib_src = [join('_mylib', 'src', x) for x in mylib_src]

    mylib_headers = []
    mylib_headers = [join('_mylib', 'src', x) for x in mylib_headers]

    mylib_dep = mylib_headers + mylib_src

    config.add_extension('_mylib',
                         sources=mylib_src,
                         depends=mylib_dep,
                         include_dirs=[join('_mylib', 'src')])
    # MODIFICATION END    

    return config

if __name__ == '__main__':
    from numpy.distutils.core import setup
    setup(**configuration(top_path='').todict())

Run scipy install

Installing scipy now (from base-dir) with:

python3 setup.py build_ext --inplace

works and we will see those shared-libraries in scipy/optimize/:

_lbfgsb.cpython-35m-x86_64-linux-gnu.so
...
_mylib.cpython-35m-x86_64-linux-gnu.so

Let's try to use it / a look into scipy/optimize/mylib.py

As seen in the other question, we can get the lib with (we are in scipy/optimize/mylib.py) and get the function we want to use:

import scipy as scp
import numpy.ctypeslib as ctl
lib = ctl.load_library('_mylib', scp.optimize.__file__)
myfunc = lib.hallo

Now here is the problem: this fails with:

AttributeError: /.../_mylib.cpython-35m-x86_64-linux-gnu.so: undefined symbol: hallo

Trying it per hand:

import ctypes
lib = ctypes.CDLL('full path to above so')
myfun = lib.hallo

fails too.

But, checking (on my linux OS): nm --defined-only _mylib.cpython-35m-x86_64-linux-gnu.so outputs:

...
...
0000000000000530 t hallo
...

which should be ok. (Disclaimer: i absolutely miss any knowledge about C++ linking). Edit: maybe it's not okay. See later observation in regards to t vs. T!

Doing it manually: this works

Going to scipy/optimize/_mylib/src:

g++ -shared -fPIC debug.cpp -o mylib.so

followed by nm --defined-only mylib.so shows the same functions, but some t became T.

...
...
0000000000000600 T hallo
...

Probably this is the reason, and there is some related general question here.

As mentioned, this works:

# in src
import numpy.ctypeslib as ctl
lib = ctl.load_library('mylib.so', '.')
lib.hallo
# < FuncPtr object at 0x....

But what needs to be done in setup.py or my sources to make it work?

During running scipy-install, output looks like:

building 'scipy.optimize._mylib' extension
compiling C++ sources
C compiler: x86_64-linux-gnu-g++ -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC

creating build/temp.linux-x86_64-3.5/scipy/optimize/_mylib
creating build/temp.linux-x86_64-3.5/scipy/optimize/_mylib/src
compile options: '-Iscipy/optimize/_mylib/src -I/usr/local/lib/python3.5/dist-packages/numpy-1.15.0.dev0+e4d678a-py3.5-linux-x86_64.egg/numpy/core/include -I/usr/include/python3.5m -c'
x86_64-linux-gnu-g++: scipy/optimize/_mylib/src/debug.cpp
scipy/optimize/_mylib/src/debug.cpp: In function ‘void hallo()’:
scipy/optimize/_mylib/src/debug.cpp:3:9: warning: unused variable ‘a’ [-Wunused-variable]
     int a = 0;
         ^
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/scipy/optimize/_mylib/src/debug.o -Lbuild/temp.linux-x86_64-3.5 -o scipy/optimize/_mylib.cpython-35m-x86_64-linux-gnu.so -Wl,--version-script=build/temp.linux-x86_64-3.5/link-version-scipy.optimize._mylib.map
sascha
  • 32,238
  • 6
  • 68
  • 110

0 Answers0