Segmentation fault when calling a more complex C++ function from Python with ctypes

Question

I have a function in C++ code which I want to call from Python with ctypes. The shared library (libRunaphys.so, I'm on Linux) contains a lot of other functions but I only need to use one function. The function is called advance_runaway_population() and it is not part of any class. I wrote the following after the function descritptions in

control.cpp

extern "C"
{
double Runaphys_advance_runaway_population(const plasma_local &plasma_local, double timestep, double inv_asp_ratio, double rho_tor_norm, module_struct const &modules, double *rate_values){return advance_runaway_population(plasma_local, timestep, inv_asp_ratio, rho_tor_norm, modules, rate_values);}
}

where plasma_local and module_struct are both custom structures and rate_values is an array of 4 doubles. My header file looks like

control.h

#ifndef CONTROL_H_
#define CONTROL_H_
    
#include <vector>
#include <string>
#include "plasma_structures.h"
    

struct module_struct {
    std::string dreicer_formula;
    bool dreicer_toroidicity;
    std::string avalanche_formula;
    bool avalanche_toroidicity;
    bool hdf5_output;
    double warning_percentage_limit;
    double rho_edge_calculation_limit;
    
};

#ifdef __cplusplus
extern "C" {
#endif

double advance_runaway_population(const plasma_local &plasma_local, double timestep, double inv_asp_ratio, double rho_tor_norm, module_struct const &modules, double *rate_values);
    
#ifdef __cplusplus
}
#endif

int list_parameter_settings(module_struct const &modules);

#endif /* CONTROL_H_ */

I wrote a Python script in which I want to test the C++ function:

python_constructor.py

import ctypes as ct
from os import path

class MODULE(ct.Structure):
    _fields_ = [("dreicer_formula", ct.c_char_p),
            ("dreicer_toroidicity", ct.c_bool),
            ("avalanche_formula", ct.c_char_p),
            ("avalanche_toroidicity", ct.c_bool),
            ("hdf5_output", ct.c_bool),
            ("warning_percentage_limit", ct.c_double),
            ("rho_edge_calculation_limit", ct.c_double)]

string1 = "string 1"
string2 = "string 2"
# create byte objects from the strings
dreicer_formula1 = string1.encode('utf-8')
avalanche_formula1 = string2.encode('utf-8')

dreicer_toroidicity = ct.c_bool(True)
avalanche_toroidicity = ct.c_bool(True)
hdf5_output = ct.c_bool(False)
warning_percentage_limit = ct.c_double(1.0)
rho_edge_calculation_limit = ct.c_double(0.85)

modules = MODULE(dreicer_formula1, dreicer_toroidicity, avalanche_formula1, avalanche_toroidicity, hdf5_output, warning_percentage_limit,rho_edge_calculation_limit)

#get the pointer to the structure
modules_pointer = ct.byref(modules)

class PLASMA(ct.Structure):
    _fields_ = [("rho", ct.c_double),
            ("electron_density", ct.c_double),
            ("electron_temperature", ct.c_double),
            ("effective_charge", ct.c_double),
            ("electric_field", ct.c_double),
            ("magnetic_field", ct.c_double),
            ("runaway_density", ct.c_double)]

rho = ct.c_double(1.8)
electron_density = ct.c_double(1e21)
electron_temperature = ct.c_double(1e5)
effective_charge = ct.c_double(1.0)
electric_field = ct.c_double(1.0)
magnetic_field = ct.c_double(2.0)
runaway_density = ct.c_double(3e15)

plasma_local = PLASMA(rho, electron_density, electron_temperature, effective_charge, electric_field, magnetic_field, runaway_density)
plasma_pointer = ct.byref(plasma_local)

basepath = path.dirname("python_constructor.py")
library_path = path.abspath(path.join(basepath, "..", "build/src/libRunaphys.so"))
lib_Runaphys = ct.CDLL(library_path)
adv_RE_pop = lib_Runaphys.Runaphys_advance_runaway_population
adv_RE_pop.restype = ct.c_double

timestep = ct.c_double(1e-3)
inv_asp_ratio = ct.c_double(0.30303)
rho_tor_norm = ct.c_double(0.65)
rate_values_type = ct.c_double * 4
rate_values = rate_values_type(0.,0.,0.,0.)
rate_values_pointer = ct.byref(rate_values)

adv_RE_pop.argtypes = [ct.POINTER(PLASMA), ct.c_double, ct.c_double, ct.c_double, ct.POINTER(MODULE), ct.POINTER(rate_values_type)]

answer = adv_RE_pop(ct.byref(plasma_local), timestep, inv_asp_ratio, rho_tor_norm, ct.byref(modules), ct.byref(rate_values))

print(answer)

I managed to make structures from Python, though I'm not sure wheter they are suitable for C++ as well. Also I found another question regarding passing strings and I made my MODULE struct accordingly.

The Python code runs fine until the C++ function call, where it exits with Segmentation fault. I've been searching and I found that my problem is most likely with the passing of pointers between Python and C++. Since my C++ function requires pretty specific structures and isn't part of a class I couldn't find a tutorial to use. I was reading the ctypes documentation too but I couldn't apply those examples to my usecase either. I primarily work in Python so I'm not too familiar with C++, C and ctypes so please point out the trivial mistakes too.

How can I pass pointers to C structures (made in Python) from Python to C++?

If possible I would like to use ctypes because the code should be as portable as possible, but if there is a much simpler way for me to be able to call this C++ function I am open to that solution as well.

https://stackoverflow.com/questions/25385173/what-is-a-debugger-and-how-can-it-help-me-diagnose-problems — πάντα ῥεῖ, Apr 18 '21 at 11:16
_"Since my C++ function requires pretty specific structures and isn't part of a class ..."_ Wrap it into a suitable one for your needs in interfacing with python. — πάντα ῥεῖ, Apr 18 '21 at 11:38
The original `C++` library is used not only by me, and has releases. I will have to merge my modifications into the main branch at some point but I would like to keep the original library as lightweight as possible. Based on this, do you still think my best option is to wrap the code in `C++`? (I don't mind writing a long Python script.) — Ferenc Lengyel, Apr 18 '21 at 11:55
Another question: How can I pass pointers to `C` structures (made in Python) from Python to `C++`? — Ferenc Lengyel, Apr 18 '21 at 12:00
I believe you should use Python's c-API to do that. To interface with c++ code, it's probably better to write wrapper code (wrapping means you don't need to _merge_ that in that original library, but bind python with your own) using some helper framework as e.g. [`boost::python`](https://www.boost.org/doc/libs/1_76_0/libs/python/doc/html/index.html). — πάντα ῥεῖ, Apr 18 '21 at 12:14
I searched more and here's [the closest thing I found](https://stackoverflow.com/questions/4351721/python-ctypes-passing-a-struct-to-a-function-as-a-pointer-to-get-back-data). I will try to modify my code based on this thread and report back. — Ferenc Lengyel, Apr 18 '21 at 14:25
*advance\_runaway\_population* arguments don't match (at least the last one). Also try to follow a coding style: https://www.python.org/dev/peps/pep-0008. Also there are lots of questions / answers here on this topic. — CristiFati, Apr 18 '21 at 15:44
I changed the last argument type to `ct.POINTER(ct.c_double)` and now I'm getting an error `expected LP_c_double instance instead of pointer to c_double_Array_4 `. If I change `rate_values1` to a simple `ct.c_double` then I once again get Segmentation fault. This leads me to believe that the problem might be that the C++ function gets the arguments succesfully, just cannot do with them what it's supposed to. Or in that case I would get another type of error? — Ferenc Lengyel, Apr 18 '21 at 16:00
`std::string` != `c_char_p` in Python. Can't use a C++ type in `"C"types`. You'll have to wrap `module_struct` as well. — Mark Tolonen, Apr 18 '21 at 16:15

score 1 · Accepted Answer · answered Apr 18 '21 at 16:55

Here's a working example. the module_struct std::string values can't be generated in Python, so a wrapper structure with C types is used to pass from Python to C, and the extern "C" function takes that wrapper class instance and converts it to a proper C++ class instance.

Also you don't have to wrap every parameter in a ctypes type. ctypes "knows" the wrapper type if .argtypes are declared correctly.

for passing rate_values, ct.byref(rate_values) is equivalent to double(*)[4] type. Don't use ct.byref and the c_double * 4 array will be passed as ct.POINTER(double). this is similar to a C array decaying to a pointer as a parameter.

control.cpp:

#include <string>

#ifdef _WIN32
#   define API __declspec(dllexport)
#else
#   define API
#endif

struct plasma_local
{
    double rho;
    double electron_density;
    double effective_charge;
    double electric_field;
    double magnetic_field;
    double runaway_density;
};

// ctypes-compatible structure
struct module_struct_wrap
{
    const char* dreicer_formula;
    bool dreicer_toroidicity;
    const char* avalanche_formula;
    bool avalanche_toroidicity;
    bool hdf5_output;
    double warning_percentage_limit;
    double rho_edge_calculation_limit;
    
};

// C++ structure
struct module_struct
{
    std::string dreicer_formula;
    bool dreicer_toroidicity;
    std::string avalanche_formula;
    bool avalanche_toroidicity;
    bool hdf5_output;
    double warning_percentage_limit;
    double rho_edge_calculation_limit;
};

// dummy C++ function to validate parameters were passed correctly.
double advance_runaway_population(
        const plasma_local &plasma_local,
        double timestep,
        double inv_asp_ratio,
        double rho_tor_norm,
        module_struct const &modules,
        double *rate_values)
{
    printf("plasma_local(%g,%g,%g,%g,%g,%g)\n",
        plasma_local.rho,
        plasma_local.electron_density,
        plasma_local.effective_charge,
        plasma_local.electric_field,
        plasma_local.magnetic_field,
        plasma_local.runaway_density);
    printf("t=%g,iar=%g,rtn=%g\n",timestep,inv_asp_ratio,rho_tor_norm);
    printf("module_struct(%s,%d,%s,%d,%d,%g,%g)\n",
        modules.dreicer_formula.c_str(),
        modules.dreicer_toroidicity,
        modules.avalanche_formula.c_str(),
        modules.avalanche_toroidicity,
        modules.hdf5_output,
        modules.warning_percentage_limit,
        modules.rho_edge_calculation_limit);
    for(int i = 0; i < 4; ++i)
        printf("rate_values[%d] = %g\n",i,rate_values[i]);
    return 1.0;
}

// ctypes-compatible wrapper function
extern "C" API
double Runaphys_advance_runaway_population(
        const plasma_local &plasma_local,
        double timestep,
        double inv_asp_ratio,
        double rho_tor_norm,
        module_struct_wrap const &modules_wrap,
        double *rate_values)
{
    // convert ctypes-compatible structure to required C++ structure.
    module_struct modules;
    modules.dreicer_formula = modules_wrap.dreicer_formula;
    modules.dreicer_toroidicity = modules_wrap.dreicer_toroidicity;
    modules.avalanche_formula = modules_wrap.avalanche_formula;
    modules.avalanche_toroidicity = modules_wrap.avalanche_toroidicity;
    modules.hdf5_output = modules_wrap.hdf5_output;
    modules.warning_percentage_limit = modules_wrap.warning_percentage_limit;
    modules.rho_edge_calculation_limit = modules_wrap.rho_edge_calculation_limit;
    return advance_runaway_population(plasma_local, timestep, inv_asp_ratio, rho_tor_norm, modules, rate_values);
}

test.py

import ctypes as ct

class MODULE(ct.Structure):
    _fields_ = [("dreicer_formula", ct.c_char_p),
                ("dreicer_toroidicity", ct.c_bool),
                ("avalanche_formula", ct.c_char_p),
                ("avalanche_toroidicity", ct.c_bool),
                ("hdf5_output", ct.c_bool),
                ("warning_percentage_limit", ct.c_double),
                ("rho_edge_calculation_limit", ct.c_double)]

class PLASMA(ct.Structure):
    _fields_ = [("rho", ct.c_double),
            ("electron_density", ct.c_double),
            ("electron_temperature", ct.c_double),
            ("effective_charge", ct.c_double),
            ("electric_field", ct.c_double),
            ("magnetic_field", ct.c_double),
            ("runaway_density", ct.c_double)]

modules = MODULE(b'string 1',True,b'string 2',True,False,1.0,0.85)
plasma_local = PLASMA(1.8,1e21,1e5,1.0,1.0,2.0,3e15)

lib_Runaphys = ct.CDLL('./control')
adv_RE_pop = lib_Runaphys.Runaphys_advance_runaway_population
adv_RE_pop.argtypes = ct.POINTER(PLASMA),ct.c_double,ct.c_double,ct.c_double,ct.POINTER(MODULE),ct.POINTER(ct.c_double)
adv_RE_pop.restype = ct.c_double

rate_values = (ct.c_double * 4)(0.,0.,0.,0.)
answer = adv_RE_pop(ct.byref(plasma_local),1e-3,0.30303,.65,ct.byref(modules),rate_values)
print(answer)

Output:

plasma_local(1.8,1e+21,100000,1,1,2)
t=0.001,iar=0.30303,rtn=0.65
module_struct(string 1,1,string 2,1,0,1,0.85)
rate_values[0] = 0
rate_values[1] = 0
rate_values[2] = 0
rate_values[3] = 0
1.0

Thank you so much for taking the time to do a complete solution! It works for me as well. — Ferenc Lengyel, Apr 18 '21 at 17:45

Segmentation fault when calling a more complex C++ function from Python with ctypes

1 Answers1