Compile python program to library importable in C

Question

I started off today wondering whether it is possible to save a python object for use in a C program, a proposition which, after many hours of reading looks naive. Here is a possible workaround:

1. Create a complex object dependent on many python libraries with data inside I need preserved.
2. Pickle the complex object and place it where it will be accessible.
3. Define compileme.py:

import pickle
thing = pickle.load(open('thing.pkl', 'r'))# an object with a method query(),
                                           # which takes a numpy array as input

4. cython --embed -o compileme.c compileme.py to generate a .c version of the script.
5. Define main.c:

#include <stdio.h>
#include//(A) something from compileme

int main(void) {
    input = //(B) query takes a numpy array in python. Define something palatable.
    double result = thing.query(input);
    printf("%d", result);
}

6. Compile main.c properly, with all the right linkages.

It is not clear to me this basic solution strategy is sound, and I have a number of concerns:

1. The thing is of a class from a library not even mentioned here, so its query() method depends on that external python. How can I ensure the relevant parts are also being compiled and linked?
2. How should I include compileme in my main.c so the thing will be accessible there? (location (A) in the code)
3. How can I appropriately define an input to thing's method here? Do I need to use one of the many types defined in compileme.c? (location (B) in the code)
4. How do I compile main.c with the proper linkages?
5. In doing all this, it appears I have to include references to the python header files from the python-dev package. Just to be clear, I am not actually including the interpreter by doing this, correct?

Here are some resources I've found during my search that prove it is possible to compile a simple python script to an executable compiled C program: Compile main Python program using Cython http://masnun.rocks/2016/10/01/creating-an-executable-file-using-cython/

Here is some relevant cython documentation: http://cython.readthedocs.io/en/latest/src/reference/compilation.html

Answer 1

I'm afraid this answer just explains why I don't think what you want is realistic, rather than offering solutions. It's worth looking at the code that Cython generates for a slightly modified compileme.pyx

cdef public get_unpickled():
    import pickle
    return pickle.load(open('thing.pkl', 'r'))

This creates a function that you can happily call from C (the signature is generated in compileme.h and is __PYX_EXTERN_C PyObject *get_unpickled(void);). The generated '.c' file containing the implementation is quite long, but the relevant section looks like:

__pyx_t_1 = __Pyx_Import(__pyx_n_s_pickle, 0, -1);
__pyx_t_2 = __Pyx_PyObject_GetAttrStr(__pyx_v_pickle, __pyx_n_s_load);
__pyx_t_3 = __Pyx_PyObject_Call(__pyx_builtin_open, __pyx_tuple_, NULL);
__pyx_t_1 = __Pyx_PyObject_CallOneArg(__pyx_t_2, __pyx_t_3);

I've cut this down quite a bit for clarity (mostly removing reference counting and some checks) but you can see it uses the Python import mechanism to load the pickle module from the Python standard library, it does getattr to get the function load. It calls the Python builtin open and then it calls pickle.load. All of these operations need libpython.

Then we consider what pickle does - it basically gets the .py file your class came from, imports that, and creates a new instance of your class, then populates the instance dictionary with data from the file (possibly calling some special methods if present). Again, this is entirely dependent on the using Python.

Finally let's consider what you can do with the result of get_unpickled. You have a PyObject*, a fairly opaque C structure. Most of its information is probably stored in its internal Python dictionary, which you can access through the Python C API PyObject_GetAttrString and related functions. However this data is still stored as other PyObjects which you will need to access using the Python C API. (If it's a Cython class the data may be stored in more accessible C struct fields which require less use of libpython, but probably not none).

---

In summary, Cython is largely implemented using the Python C API, which requires access to the libpython library for anything but the absolute most trivial programs. Using Python standard library functions such as pickle requires the Python standard library is installed too. Therefore you can't really achieve this without needing to bundle Python with your C program. The examples that you linked fall in this category - they are C programs but they depend on Python being present.

---

A better solution might be to look at common serialization formats that both Python and C support, such as JSON, XML, or HDF5 to allow you to save the data in one language and retrieve it in the other with as little effort as possible.