2

There is clearly something I'm not doing correctly in my use of symbol versions, but I can't put my finger on what it is.

Background: I'm working on an application that needs to be linked to Python 2.7, but it also uses a third-party plugin that requires Python 2.6. Spare me the lecture about Python 2 being obsolete: I know. But it's a non-negotiable requirement for this project. Pretend the libraries are something else if you prefer.

An earlier version of my question was put here. Since the last activity on that question, I've:

  • Established that this issue doesn't exist in Windows; it's purely on Linux, where for the record I'm currently using CentOS 7.
  • Recompiled libpython2.7.so.1.0 with the -fno-semantic-interposition compiler flag and --default-symver linker flag.
  • Verified that libpython2.7.so.1.0 contains the required symbol versions, viz.
> objdump -T libpython2.7.so.1.0 | fgrep Py_
00000000000e8920 g    DF .text  000000000000007a  libpython2.7.so.1.0 _Py_With
00000000000e8490 g    DF .text  000000000000009a  libpython2.7.so.1.0 _Py_ClassDef
0000000000042900 g    DF .text  00000000000000fd  libpython2.7.so.1.0 _Py_addlabel
00000000000e8a60 g    DF .text  000000000000004d  libpython2.7.so.1.0 _Py_TryFinally
00000000000e8b70 g    DF .text  0000000000000059  libpython2.7.so.1.0 _Py_ImportFrom
0000000000412b74 g    DO .bss   0000000000000004  libpython2.7.so.1.0 Py_InteractiveFlag
etc.
  • Verified that, after compiling and linking, my own executable knows to look for the correct versions of the symbols
> objdump -T myApplication | fgrep Py_
0000000000000000      DF *UND*  0000000000000000  libpython2.7.so.1.0 Py_Finalize
0000000000000000      DO *UND*  0000000000000000  libpython2.7.so.1.0 Py_NoUserSiteDirectory
0000000000000000      DF *UND*  0000000000000000  libpython2.7.so.1.0 Py_CompileStringFlags
0000000000000000      DF *UND*  0000000000000000  libpython2.7.so.1.0 Py_IsInitialized
0000000000000000      DF *UND*  0000000000000000  libpython2.7.so.1.0 Py_Initialize
0000000000000000      DF *UND*  0000000000000000  libpython2.7.so.1.0 Py_DecRef
0000000000000000      DF *UND*  0000000000000000  libpython2.7.so.1.0 Py_SetPythonHome
0000000000000000      DO *UND*  0000000000000000  libpython2.7.so.1.0 Py_IgnoreEnvironmentFlag

Tried additionally inserting __asm(".symver ...") declarations into my source file as suggested for example here:

#include <Python.h>

__asm__(".symver Py_NoUserSiteDirectory,Py_NoUserSiteDirectory@libpython2.7.so.1.0");
__asm__(".symver Py_CompileStringFlags,Py_CompileStringFlags@libpython2.7.so.1.0");
__asm__(".symver Py_SetPythonHome,Py_SetPythonHome@libpython2.7.so.1.0");
__asm__(".symver Py_Initialize,Py_Initialize@libpython2.7.so.1.0");
__asm__(".symver Py_IsInitialized,Py_IsInitialized@libpython2.7.so.1.0");
__asm__(".symver Py_DecRef,Py_DecRef@libpython2.7.so.1.0");
__asm__(".symver Py_IgnoreEnvironmentFlag,Py_IgnoreEnvironmentFlag@libpython2.7.so.1.0");
__asm__(".symver Py_Finalize,Py_Finalize@libpython2.7.so.1.0");

Yet after all that:

  1. I have to preload libpython2.6.so.1.0 or else the plugin, when it's invoked, calls the Python 2.7 versions of the functions and crashes.
    export LD_PRELOAD=<path to>/libpython2.6.so.1.0
  1. Subsequent calls to Python functions from my own code lead to the Python 2.6 versions being invoked:
    import sys
    for p in sys.path:
        print p

    ../lib/python26.zip
    ../lib/python2.6/
    ../lib/python2.6/plat-linux2
    ../lib/python2.6/lib-tk
    ../lib/python2.6/lib-old
    ../lib/python2.6/lib-dynload

So can anyone tell me what I'm missing here?

If all else fails, I'll make some more invasive changes to the Python 2.7 source code, for example aliasing the public functions and relying on the -fno-semantic-interposition flag to ensure that they call the correct unaliased functions, viz.

Python.h
--------

PyAPI_FUNC(void) MyPy_SetPythonHome(char *);  


MyPython.c
----------

PyAPI_FUNC(void) MyPy_SetPythonHome(char * path)
{
    Py_SetPythonHome(path);
}

...but this smells really bad and I'd rather avoid it.

Rachid K.
  • 4,490
  • 3
  • 11
  • 30
Eos Pengwern
  • 1,467
  • 3
  • 20
  • 37
  • If I understand well, your "own code" is written in C and you want to make sure that it will call the python2.7 functions instead of python2.6 ? If the answer is yes, I had to solve a similar problem in the past. At program startup, I used dlvsym() for each services to get the address of the API corresponding to the version I needed. Hence, I had a set of function pointers to the desired version and I used them to call the API. – Rachid K. Oct 22 '20 at 12:36
  • Yes, your understanding is correct. I'm been really hoping to find a way of doing this that avoids "dlopen" etc. because a lot of the Python invocations are made from code auto-generated by SWIG and Shiboken. Even getting them to use alternative names for the Python functions will be awkward enough, but getting them to replace all their function invocations with pointers will be harder still. Hence I'd like to try to get the linker to do the work if I *possibly* can. – Eos Pengwern Oct 22 '20 at 13:59
  • 1
    What about overloading the symbols in your "own code" ? But those overloaded function call the real ones through the pointer got from dlvsym(). For example: you redefine Py_SetPythonHome(...) { static int first_call = 1; static void *func; if (first_call) { func = dlvsym(RTLD_DEFAULT, "Py_SetPythonHome", "PYTHON_2.7"); first_call = 0;} call func(...); }. So, you don't need to rename the API in the auto-generated code and the latter will link with your overloaded API. I may have forgotten some details. Just trying to help... – Rachid K. Oct 22 '20 at 14:40
  • Now that's a very good idea; I think I can make that work. I'll have a go, anyway.... – Eos Pengwern Oct 22 '20 at 18:49

0 Answers0