0

I am trying to make an interface between c and python. And by suing interface i want to call python user defined methods from c for some specific operations. I am using Cython for interface. Given a python module, i am compiling using cython and generating dynamic library (.so file). Then that library is being used to compile c driver program to make use of python module.

So when i am doing this from particular virtual environment where the third-party module is installed (used in module). Its working fine. But when i do the same thing outside the environment where python module doesn't have third-party module its failing.

Description of code and procedure does as follows;

pyjlib.pyx (cython module where python functions are defined)

import ujson as json

cdef unicode cstring2unicode(char* s):
    if s == NULL:
        return None
    else:
        return s.decode("UTF-8", "replace")


cdef char * unicode2cstring(unicode s):
    return s.encode("UTF-8")

cpdef public char * get_ouput(char * cargstr):
    cdef unicode argstr = cstring2unicode(cargstr)
    cdef dict argdict = json.loads(argstr)
    argdict["output"]="cool"
    outstr = json.dumps(argdict)
    cdef unicode retstr = outstr.decode('UTF-8')
    return unicode2cstring(retstr)

In the this module i am using ujson, which is a third party module to python distribution.

compile.py (script to generate .so file)

from distutils.core import setup, Extension

module = Extension('pyjlib', ['pyjlib.c'])


setup(
    name='pyjlib',
    version='1.0',
    ext_modules=[module]
)

pyjlib_test.c (c driver program where python module is being used;)

#include "Python.h"
#include "pyjlib.h"
#include <stdio.h>

int main(int argc, char const *argv[]) {
    char * instr = "{\"name\":\"Jai\"}";
    Py_Initialize();
    initpyjlib();
    char * outstr = get_ouput(instr, 0);
    Py_Finalize();

    printf("input : %s\n\n",instr );
    printf("output : %s\n",outstr );
    return 0;
}

To generate dynamic library below procedure is being followed. Below procedure i used to in a python virtual environment where ujson is installed.

# using cython generate pyjlib.c and pyjloib.h from pyjlib.pyx

cython pyjlib.pyx

# compile .c file to generate .so file

python compile.py build_ext --inplace

# make .so file c compiler/linker recognisable

ln -s pyjlib.so libpyjlib.so

after doing above procedure, i will have pyjlib.so and libpyjlib.so file.

# c driver program compilation

gcc -fPIC -o pyjlib_test pyjlib_test.c -I ./ -I /usr/include/python2.7 `python-config --libs` -L ./ -lpyjlib -lpython2.7

After doing this i will have executable 'pyjlib_test'

If i run the executable inside same python virtualenv then i get CLI output as follows;

./pyjlib_test


input : {"name":"Jai"}

output : {"output":"cool","name":"Jai"}

But if I run it outside the virtualenv, CLI output goes as follows;

./pyjlib_test 


NameError: name 'json' is not defined
Exception NameError: "name 'json' is not defined" in 'pyjlib.get_ouput' ignored
input : {"name":"Jai"}

output : (null)

Which means i think, during the compilation python thrird party modules are included or defined in .so file.

*So can anyone please help how to include / bundle python third-party module to .so file so that it can be accessible from outside python environment. *

Thank you.

Biswajit
  • 59
  • 6
  • What it means is that `initpyjlib()` has raised an exception (probably an `ImportError`) which you aren't checking. You can investigate this with `PyErr_Occurred()` and `PyErr_Print()`. That will tell you the actual first exception that happens, which will be more informative. The chances are you'll just have to bundle `json` with your module. – DavidW Apr 18 '20 at 17:15
  • @DavidW Thanks for the reply. If you know how to bundle any third-party python package to .so file at compilation time. Can you please help me out in this. – Biswajit Apr 19 '20 at 02:09
  • You didn't read may be what i wrote. I know clearly its import error. Only the thing is i have not included `PyErr_Occurred()` , `PyErr_Print()` and other debugging details over here. I have confirmed it already. what I am asking is, how to bundle third party module. here Its not json exactly, its ujson. i am importing ujson as json so its giving me undefined json because its unable to import the module outside the scope. – Biswajit Apr 19 '20 at 02:10
  • One option would be to compile json/ujson with Cython then follow the instructions here https://stackoverflow.com/questions/52959902/make-executable-file-from-multiple-pyx-files-using-cython. However I really don't recommend this - it isn't what Cython is designed for, the chances are json contains multiple submodules and had other dependencies and you'd have to handle all of them. – DavidW Apr 19 '20 at 06:37
  • It'd probably be better though just to zip up your dependencies and distribute them with your exe. You can set up your Python path in main. – DavidW Apr 19 '20 at 07:36
  • I know the actual usage of cython. This is my short time need. So doing in this way. So anyways, you mean to say, if i can compile whole dependency modules and add them into library? That will be tough job i think, if library is a big library like numpy. – Biswajit Apr 19 '20 at 11:17
  • You can definitely compile _simple_ dependency modules and add them into the library. However it will be much harder for big packages like Numpy which are made up of their own compiled modules and data files. I don't know if anyone has actually managed to do this, so I really can't recommend it. Everything I know about the subject is in the question I linked to, which is only really a simple demo with two modules. – DavidW Apr 19 '20 at 12:40
  • OK. Thank you @DavidW. Let me check it. If i get any alternative i will let you know too. :) – Biswajit Apr 19 '20 at 12:52

0 Answers0