How can I compile multiple C++ files to use them with Python ctypes?

Question

I have a little problem with the compilation of multiple C++ on Windows. I implemented four classes in C++ for cryptography with gmp. I want to call them from Python with ctypes. I wrote a cpp file with the extern keyword:

#include "integer.h"
#include "modular_number.h"
#include "padic_number.h"
#include "rational_number.h"

extern "C" {
    __declspec(dllexport) ModNum* newModNum(const char * n, const char * p) { return new ModNum(Integer(n), Integer(p)); }
    __declspec(dllexport) const char* getModValue(const ModNum& mod){ return mod.getValue().getValue(); }

    __declspec(dllexport) RationalNum* newRationalNum(const char* mpq) { return new RationalNum(mpq); }
    __declspec(dllexport) const char* getRationalValue(const RationalNum& rat){ return rat.getValue(); }

    __declspec(dllexport) PadicNum* newPadicNum(const char* n, const char* base) { return new PadicNum(Integer(n), Integer(base)); }
    __declspec(dllexport) const char* getPadicValue(const PadicNum& padic){ return padic.getValue().getValue(); }
}

I compiled my files with:

mingw32-g++ -fexceptions -g -fexpensive-optimizations -flto -O3 -Weffc++ -Wextra -Wall -std=c++14 -fPIC -Og -IC:\MinGW\include -flto -s -lgmp -lmpfr -lpthread -c -fPIC *.cpp -I"C:\Program Files\Python38-32\include" -I"C:\Program Files\Python38-32\libs"

mingw32-g++.exe -shared -Wl,-dll -o numeric.dll *.o -lgmp -lmpfr -lgmpxx -static

But when I use these commands in Python:

import ctypes;
x = ctypes.DLL("./numeric.dll");

The variable x does not have the functions: newModNum, getModValue, etc... Could anyone tell me what I'm doing wrong? I get no error and I do not understand. My other files are common C ++ files with header and implementation.

Thanks in advance and have a nice day!

Have you referenced them once (e.g. , `x.newModNum`)? `ctypes` loads a function on first use. `dumpbin /exports numeric.dll` can show you what was really exported. Assuming you can find dumpbin.exe. Its usually available if you have visual studio and use the visual studio command prompt. — tdelaney, Feb 12 '20 at 17:42
Another tool to add to your box is `dependencywalker` (http://dependencywalker.com), which has helped me a lot in the past with DLLs and exports. BTW: You're supposed to provide a [mcve]. In your case, you would have been able to eliminate GMP and "multiple" from the problem space, reducing the remaining complexity. — Ulrich Eckhardt, Feb 12 '20 at 18:41
@tdelaney No, I did not refer them because I thought that ctypes looks for `extern` keyword. I saw a lot of examples and I did not notice that others referred their function to be exported. Just like in this post https://stackoverflow.com/questions/145270/calling-c-c-from-python. — G.F, Feb 12 '20 at 20:17
@G.F - `ctypes` loads on demand. For instance, in one of the answers to your referenced question https://stackoverflow.com/a/145649/642070, the first `self.obj = lib.Foo_new()` loads `Foo_new`. you could just put a `x.newModNum; newModNum in dir(x)` to check. BTW, you will likely want to add `argtypes` and `restype` type hints to your functions before you use them and this conveniently is also the first reference that loads them. — tdelaney, Feb 12 '20 at 21:02
@tdelaney So, even if I didn't see any of my external functions in `dir(x)`, if I write x.newModNum there will be an automatic loading of the function? — G.F, Feb 12 '20 at 21:17
@G.F- Yes! Just add `x.newModNum` and see if it works. I showed an example with the standard libc below. — tdelaney, Feb 12 '20 at 21:19
As an aside, its easy for `ctypes` integrations to go very wrong. Back when I was doing C callouts, I found the python C extension interface to be a much safer way to do things. In addition to the documentation, there are many examples in the python source itself to review. https://docs.python.org/3/extending/extending.html#extending-python-with-c-or-c — tdelaney, Feb 12 '20 at 21:29
@tdelaney Why is it much safer than use directly C or C++ code? The code should be faster with pure C or C++. Why should it be unsafe? Anyway, I must use this strategy because I need GMP and gmpy2 is not so fast when numbers become very huge (more than 1000 digits). I tried to use Cython but up to now it is not possible for Windows but only for Linux. — G.F, Feb 12 '20 at 21:39
With `ctypes` you loose the native types (e.g., `ModNum*` may become a python `c_void_p` or an `int`) and later uses such as `ModNum&` or maybe some `ModNum**` list down the line can be awkward to get right in ctypes. you've got a thin and well-contained interface here, but if it starts growing and its use spreads throughout a large code base, you add a lot of late night debugging sessions to your life. — tdelaney, Feb 12 '20 at 21:52
Ah, ok! I will see how it works and in case I will try to convert my code using these tips https://docs.python.org/3/extending/extending.html#extending-python-with-c-or-c — G.F, Feb 12 '20 at 22:15

score 1 · Accepted Answer · answered Feb 12 '20 at 21:14

1

ctypes functions are imported on first use. Using libc as an example:

>>> import ctypes
>>> libc = ctypes.CDLL("libc.so.06")
>>> "printf" in dir(libc)
False
>>> libc.printf
<_FuncPtr object at 0x7f6512c23430>
>>> "printf" in dir(libc)
True

ctypes assumes all parameters and the return value are int. You should give type hints which also conveniently import the functions.

import ctypes
x = ctypes.DLL("./numeric.dll")
x.newModNum.argtypes = [ctypes.c_char_p, ctypes.c_char_p] # <-- also imports
x.newModNum.rettype = ctypes.c_void_p

And remove the semicolons from the end of lines. It causes dangerous blood pressure spikes in python programmers.

answered Feb 12 '20 at 21:14

tdelaney

73,364
6
83
116

O.o Did I write the semicolon? I did not realize it o.O Anyway thanks for your help! Now I cannot try because I don't have the code with me. I will try tomorrow morning and if have any problem I will write it. – G.F Feb 12 '20 at 21:27
Ok, it works, but as you said it does not returns an object but an integer. So I can't use it... Is there no way to call a C++ function an use its objects like python objects without implement a PyObject? Do I must implement a function for each operation? – G.F Feb 13 '20 at 08:22
In the end, I implemented an `extern` function for each arithmetic operation. Now I can take advantage of C++ speed for this operations. Anyway, thanks a lot for your help! It was very precious! – G.F Feb 13 '20 at 09:19

How can I compile multiple C++ files to use them with Python ctypes?

1 Answers1