7

Many well-known python libraries are basically written in C (like tensorflow or numpy) because this apparently speeds things up a lot. I was able to very easily integrate a C function into python by reading this. Doing so I can finally use distutils to access the functions of the source.c file:

# setup.py

from distutils.core import setup, Extension

def main():
    setup(
        # All the other parameters...
        ext_modules=[ Extension("source", ["source.c"]) ]
        )

if __name__ == "__main__":
    main()

so that when i run python setup.py install i can install my library. However, what if i want to create a python-coded wrapper object for the functions inside source.c? Is there a way to do this without polluting the installed modules?
Wandering around the internet I have seen some seemingly simple solutions using shared libraries (.so). However I would need a solution that does not involve attaching the compiled C code, but one that compiles it the first time the program is run.

Giuppox
  • 1,393
  • 9
  • 35

1 Answers1

1

The shared libraries are the right way to go in this case. The distutils have ability to build the static libraries as follows:

from distutils.ccompiler import new_compiler
from distutils import sysconfig

c = new_compiler()
workdir = "."
c.add_include_dir( sysconfig.get_python_inc() )
c.add_include_dir("./include")
objects = c.compile(["file1.c", "file2.c"])
c.link_shared_lib(objects, "mylibrary", output_dir=workdir)

This will generate the .so library in the working directory.

For example how it's used in real setup see the following example

jordanvrtanoski
  • 5,104
  • 1
  • 20
  • 29
  • hi thanks for answering my question. What are `c.add_include_dir("./include")` and `c.link_shared_lib(objects, "mylibrary", output_dir=workdir)` supposed to do? Also, i tried to execute your code, and it generates me two files, one `.so` and one `.o`, what's the difference? – Giuppox Mar 08 '21 at 08:44
  • The `.o` object is the intermediate binary object file that compiler generates. The `.so` is the shared library. For each `.c` file you will have a `.o` file generated, and than the files will be combined in a library. After the files are combined, you can remove the `.o`. – jordanvrtanoski Mar 08 '21 at 08:55
  • 1
    ok thanks, what about "c.add_include_dir("./include")"? – Giuppox Mar 08 '21 at 09:00
  • This one is if you have special header files of other files you want to include in the build. Think of this as an python version of a `Makefile`, so I added the line to help someone who has the need to include additional files. – jordanvrtanoski Mar 08 '21 at 10:15
  • 1
    is there a way to make the `.o` files be generated in a directory different from their `.c` files? – Giuppox Mar 08 '21 at 11:27
  • hi, i the compiling works just fine, but when i try to `#include ` clang (that i suppose is the default compiler of distutils) raises `fatal error: 'Python.h' file not found`, what should i do? – Giuppox Mar 08 '21 at 17:59
  • You need to have python in the include. Also python development packages should be installed. If you still have issue, share the snipped of the code so I can check what you are doing. – jordanvrtanoski Mar 08 '21 at 18:23
  • 1
    the project is here https://github.com/Giuppox/block , the thing is not that complicated, i just use `setup.py` to compile `block/core.c` (where i try to `#include `) inside a generated folder `block/build`. Sorry for the huge amount of questions, and thanks for your help! – Giuppox Mar 08 '21 at 18:54
  • @Giuppox I have send pull request with the fix. Check your repository – jordanvrtanoski Mar 09 '21 at 11:28
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/229683/discussion-between-jordanvrtanoski-and-giuppox). – jordanvrtanoski Mar 09 '21 at 11:30
  • 1
    hi it is also possible to do something similar using setuptools? i was reading some articles and it seems like it is the standard now... – Giuppox Mar 13 '21 at 15:44
  • It looks like you can by calling shell commands. This is what I [found](https://stackoverflow.com/questions/33168482/compiling-installing-c-executable-using-pythons-setuptools-setup-py), but I never used this before. If I was doing the attached example, I would have used `distutils` to compile instead of using shell commands. – jordanvrtanoski Mar 17 '21 at 11:17