compile official doc "Embedding Python in Another Application" example failed

Question

I am trying to compile and run the example from https://docs.python.org/3/extending/embedding.html#very-high-level-embedding , but failed.

My environment is Ubuntu 20.04.2 LTS, with system shipped python3.8(statically built), libpython3-dev and libpython3.8-dev packages installed.

What I've tried:

main.c :

#define PY_SSIZE_T_CLEAN
#include <Python.h>

int
main(int argc, char *argv[])
{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();
    PyRun_SimpleString("from time import time,ctime\n"
                       "print('Today is', ctime(time()))\n");
    if (Py_FinalizeEx() < 0) {
        exit(120);
    }
    PyMem_RawFree(program);
    return 0;
}

From https://docs.python.org/3/extending/embedding.html#compiling-and-linking-under-unix-like-systems, get gcc flags.

tian@tian-B250M-Wind:~$ python3-config --cflags
-I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall
tian@tian-B250M-Wind:~$ python3-config --ldflags
-L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

(I don't know why python3-config output has some duplicated values, that's not a typing mistake)

gcc {copy cflags output} -o main main.c /path_to_libpython/libpython3.8.a {copy ld flags output}:

gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

make test1 gives error, nearly all related to -fPIE error:

/usr/bin/ld: /tmp/ccdMZ2Yk.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(obmalloc.o): relocation R_X86_64_32 against `.text.hot' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(initconfig.o): relocation R_X86_64_32 against symbol `_PyRuntime' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pathconfig.o): relocation R_X86_64_32 against symbol `_Py_path_config' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(preconfig.o): relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pylifecycle.o): relocation R_X86_64_32 against symbol `_PyRuntime' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pystate.o): relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pythonrun.o): relocation R_X86_64_32 against symbol `_PyParser_Grammar' can not be used when making a PIE object; recompile with -fPIE
/usr/bin/ld: /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a(pytime.o): relocation R_X86_64_32S against symbol `PyFloat_Type' can not be used when making a PIE object; recompile with -fPIE
.....

I saw there is a /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8-pic.a and tried link it instead but also failed (log).

Later I tried a Docker image(python:3.10.5-bullseye) with share build python and succeeded.

root@tian-B250M-Wind:/# python3-config --cflags
-I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall
root@tian-B250M-Wind:/# python3-config --ldflags
 -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

gcc -I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm -lpython3.10

(I add -lpython3.10 in the end)

It compiles and ./main :

root@tian-B250M-Wind:/# ./main
Today is Thu Jul 14 10:39:20 2022

What's wrong with my compilation for the Ubuntu system shipped, python3.8 one?

Anyway, I just want to validate that I can link the static python library libpythonx.y.a . So if anyone can make that work on a fresh installed machine or with non system shipped python (e.g. self-built static python), I'd like to try.

Hi Rick, did you update the system's Python 3.8 to latest (3.8.10)? Maybe there is some issue with the Ubuntu-shipped version. Can you give it a try and report back, please? — IamAshKS, Jul 20 '22 at 09:39
Also, can you add `-no-pie` to the CFLAGS (`{copy cflags output}`) and try again, please? — IamAshKS, Jul 20 '22 at 09:43
@IamAshKS Still errors. This is what I got after adding `-no-pie`. https://pastebin.com/8QFuLHR4 . Anyway, I just want to validate the process that I can link the static python library `libpythonx.y.a`. So if you can make that work on any fresh installed machine, I'd like to create a VPS and try. — Rick, Jul 20 '22 at 09:55
@IamAshKS `-no-pie` is correct, but for system ship python, I need to link `-lexpat -lz` 2 extra libraries. Thanks bro — Rick, Jul 24 '22 at 14:12

pygeek · Answer 1 · 2022-07-24T20:02:03.920

Problem

Print the output for gcc -v. What you will find within "Configured With" is the flag --enable-default-pie. PIE (Position Independent Executable) is enabled by default.

You are including a pre-compiled object that was not built with PIE enabled (python 3.8), therefore PIE must be disabled to compile your code if you choose to use the installed Python on your system.

The reason why you didn't encounter this in Docker is because you were building a different version of pre-compiled python 3.10, which does have PIE enabled.

Solution 1

Either, add flag -no-pie to LDFlags in the Makefile;

Or, pass -no-pie to gcc during compile time.

Solution 2

Compile your desired Python on your system with PIE flag enabled, then compile your code using the new Python you compiled with PIE.

References

https://docs.python.org/3/extending/embedding.html#very-high-level-embedding https://www.redhat.com/en/blog/position-independent-executables-pie https://linuxtut.com/en/4fc41123ed41cf443a6b/

`-no-pie` works but `-fno-pie` doesn't work. – Rick Jul 24 '22 at 14:21 — Rick, Jul 24 '22 at 14:21
@Rick I updated my answer for -no-pie on gcc as well. – pygeek Jul 24 '22 at 20:03 — pygeek, Jul 24 '22 at 20:03

Rick · Accepted Answer · 2022-07-24T14:34:09.960

I made more experiments on both system shipped python and self built python libraries. For self bulilt pythons, it's easy to compile successfully, either for shared built python or statically built python.

For system shipped python, shared link is easy and what I've missed for static link is that I need to link 2 extra libs -lexpat and -lz. Also there are 2 ways for system shipped python to link successfully:

Add -no-pie, and -lexpat -lz.
No -no-pie, use libpython3.8-pic.a, and -lexpat -lz.

Check the py38_system_static target below.

makefile:

py38_system_failed:
    gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -g -fdebug-prefix-map=/build/python3.8-4wuY7n/python3.8-3.8.10=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector -Wformat -Werror=format-security  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm

py38_system_shared:
    # 36K main
    gcc -I/usr/include/python3.8 -I/usr/include/python3.8  -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lpython3.8 -lcrypt -lpthread -ldl  -lutil -lm -lm 

py38_system_static:
    # 5.5M main 
    gcc -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -no-pie -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl  -lutil -lm -lm -lexpat -lz
    gcc -I/usr/include/python3.8 -I/usr/include/python3.8 -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c /usr/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.8-pic.a -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lcrypt -lpthread -ldl  -lutil -lm -lm -lexpat -lz

docker_py310_share:
    # 21K main
    gcc -I/usr/local/include/python3.10 -I/usr/local/include/python3.10  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/usr/local/lib  -lcrypt -lpthread -ldl  -lutil -lm -lm -lpython3.10

py36_static:
    # 15M main
    gcc -I/home/tian/py3.6.12_static/include/python3.6m -I/home/tian/py3.6.12_static/include/python3.6m  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.6.12_static/lib/python3.6/config-3.6m-x86_64-linux-gnu -L/home/tian/py3.6.12_static/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic

py36_shared:
    # 36K main
    gcc -I/home/tian/py3.6.12_shared/include/python3.6m -I/home/tian/py3.6.12_shared/include/python3.6m  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.6.12_shared/lib -lpython3.6m -lpthread -ldl  -lutil -lm  -Xlinker -export-dynamic

py39_static:
    # 22M main
    gcc -I/home/tian/py3.9.13_static/include/python3.9 -I/home/tian/py3.9.13_static/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c  -L/home/tian/py3.9.13_static/lib/python3.9/config-3.9-x86_64-linux-gnu -L/home/tian/py3.9.13_static/lib  -lpython3.9 -lcrypt -lpthread -ldl  -lutil -lm -lm 
    gcc -I/home/tian/py3.9.13_static/include/python3.9 -I/home/tian/py3.9.13_static/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main2 main.c /home/tian/py3.9.13_static/lib/libpython3.9.a -lcrypt -lpthread -ldl  -lutil -lm -lm 
    
py39_shared:
    # 36K main
    gcc -I/home/tian/py3.9.13_shared/include/python3.9 -I/home/tian/py3.9.13_shared/include/python3.9  -Wno-unused-result -Wsign-compare  -DNDEBUG -g -fwrapv -O3 -Wall -o main main.c -L/home/tian/py3.9.13_shared/lib  -lpython3.9 -lcrypt -lpthread -ldl  -lutil -lm -lm 

clean:
    rm main && rm main2

I still recommend compiling python on your system with PIE security flag enabled as outlined in solution 2 of my answer. — pygeek, Jul 24 '22 at 20:05
@pygeek Ya, I would definitely not use the system shipped python for production, as you can see there are some quirks for system shipped python. I guess those self built pythons have PIE enabled. — Rick, Jul 25 '22 at 03:56

compile official doc "Embedding Python in Another Application" example failed

2 Answers2

Problem

Solution 1

Solution 2

References

Linked