Python C_API with GCC - Python.h not found

Question

I am testing the Python C_API using the program shown below with Python 3.8 on Ubuntu 18.04. I have tried compiling with Clang-10 and GCC 7.5, but in both cases the compiler is unable to locate Python.h.

When I run this string:

sudo gcc -v -shared -fPIC -l /usr/include/python3.8 -I/usr/lib -I/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -I/usr/lib/python3.8 -lpython3.8 -lcrypt -lpthread -ldl  -lutil -lm  -Wall -o Python_Math.o Python_Math.c

I get "Python_Math.c:2:10: fatal error: Python.h: No such file or directory #include <Python.h>"

I changed the string to this:

sudo gcc Python_Math.c -o Python_Math.o -v -shared -fPIC -l/usr/lib -l/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -I/usr/include/python3.8 -l/usr/lib/python3.8 -lpython3.8 -lcrypt -lpthread -ldl  -lutil -lm -Wall

and I get:

/usr/bin/ld: cannot find -l/usr/lib /usr/bin/ld: cannot find -l/usr/lib/python3.8/config-3.8-x86_64-linux-gnu /usr/bin/ld: cannot find -l/usr/lib/python3.8

So I reduced the string to this:

sudo gcc Python_Math.c -o Python_Math.o -v -shared -fPIC -I/usr/include/python3.8 -lpython3.8 -lcrypt -lpthread -ldl  -lutil -lm -Wall

Now the program compiles and links but when I run "nm" on Python_Math.o it shows all the imported symbols (including Py_Initialize) as undefined.

Finally I reduced the string to this:

sudo gcc Python_Math.c -o Python_Math.o -v -shared -fPIC -I/usr/include/python3.8 -lpython3.8 -Wall

and again it compiles and links but nm still shows symbols as "undefined."

If I change "I" in the string above to "L" for -L/usr/include/python3.8 I again get "Python.h" not found, and a message re the search paths for includes, which shows usr/include in the path.

#include <...> search starts here:
 /usr/lib/gcc/x86_64-linux-gnu/7/include
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.

The problem appears to be the path to Python.h. Locate says Python.h is located in /usr/include/python3.8/Python.h.

A question at Embedding Python in C, linking fails with undefined reference to `Py_Initialize' said to add "embed" on Ubuntu 20.04, but I haven't found a way to make that work with Ubuntu 18.04.

Here is the complete program:

#define PY_SSIZE_T_CLEAN
#include <Python.h>

int Lib_Math(char* module_name, char* fn_name, double value_in)
{
    PyObject *pName, *pModule, *pFunc;
    PyObject *pArgs, *pValue;
    int i = 0;

    Py_Initialize();
    pName = PyUnicode_DecodeFSDefault(module_name);

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        pFunc = PyObject_GetAttrString(pModule, fn_name);
        /* pFunc is a new reference */

        if (pFunc && PyCallable_Check(pFunc)) {
            pArgs = PyTuple_New(value_in);
            pValue = PyLong_FromDouble(value_in);

            if (!pValue) {
                Py_DECREF(pArgs);
                Py_DECREF(pModule);
                fprintf(stderr, "Cannot convert argument\n");
                return 1;
            }

            /* pValue reference stolen here: */
            PyTuple_SetItem(pArgs, i, pValue);

            pValue = PyObject_CallObject(pFunc, pArgs);
            Py_DECREF(pArgs);
            if (pValue != NULL) {
                printf("Result of call: %ld\n", PyLong_AsLong(pValue));
                Py_DECREF(pValue);
            }
else {
                Py_DECREF(pFunc);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr,"Call failed\n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred())
                PyErr_Print();
            fprintf(stderr, "Cannot find function \"%s\"\n", fn_name);
        }    
        Py_XDECREF(pFunc);
        Py_DECREF(pModule);
    }
    else {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", module_name);
        return 1;
    }
    if (Py_FinalizeEx() < 0) {
        return 120;
    }
    return 0;
}

Thanks for any help with getting the right link options.

The symbols will be provided by libpython.so, use ldd to see the dependency, or you could enforce the linking against the static version, see e.g. https://stackoverflow.com/q/48796014/5769463 — ead, Oct 09 '21 at 20:27
Locate libpython.so finds nothing, but libpython3.8.so is found, so I'll work with that. — RTC222, Oct 09 '21 at 20:34

Davide Madrisan · Answer 1 · 2021-10-09T21:45:19.687

0

You messed up the I and L/l options. The right syntax is:

gcc -v -shared -fPIC \
    -I/usr/include/python3.8 \
    -L/usr/lib -L/usr/lib/python3.8 \
    -lpython3.8 -lcrypt -lpthread -ldl -lutil -lm \
    -Wall -o Python_Math.so Python_Math.c

-I for include dirs, -L for libraries dirs and -l for library files.

The compiler and linker options can be obtained by running the commands:

python3.8-config --includes
  -I/usr/include/python3.8 -I/usr/include/python3.8

python3.8-config --ldflags
  -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

You compiles the module as shared library object *.so, so objects like Py_Initialize will be available at run time.

edited Oct 09 '21 at 21:45

answered Oct 09 '21 at 19:56

Davide Madrisan

1,969
2
14
22

Your string compiles, but all the symbols from Python.h still show as "Undefined" when I do nm Python_Math.o in Ubuntu. – RTC222 Oct 09 '21 at 20:00
In the verbose version /usr/include/python3.8 shows as the first entry in the "include path" section, so GCC searches that folder, but I'm mystified why the symbols show as "Undefined." – RTC222 Oct 09 '21 at 20:02
I ran "nm" on libpython3.8 and the missing symbols are there. Now just to link against it. – RTC222 Oct 09 '21 at 20:56

Python C_API with GCC - Python.h not found

1 Answers1