dlopen() of a shared library results in multiple singletons

Question

I have application A that links statically to lib B and C.

I have dynamic library D that links statically to lib B and C and dynamic library E.

A loads D successfully with dlopen().

File-scoped variables in lib D that are classes have their constructors run when the library is opened, as expected. These constructors register themselves with a singleton factory in lib B that they find with an Instance() method.

Then app A looks for these objects with the factory, and doesn't find them.

It turns out that the singleton in the lib B inside of lib D is at a different address of the lib B inside of app A.

In other words, it's no longer a singleton.

However, if I remove lib B from the link line for dynamic lib D, lib D links fine but dlopen() fails and dlerror() reports:

 libD.so: undefined symbol: _ZN9Foo312Bar12MyFuncEd

and this symbol is a C++ method defined in lib B.

Question: should be obvious, but, can I build the app somehow so its copy of lib B is visible to the dlopen()'d lib D?

I'm running on an Intel 64-bit CPU with Fedora31.

I will be facing the exact same issues on Win10/11 as soon as this is working on Linux.

Seems like a design problem. Why are the static libs included in the app and a *.dll* that is used by the app? — CristiFati, Oct 04 '22 at 08:24
Library B is a general library with for instance logging functions. The app needs to log status and warnings and errors. So does the .dll D. Library C is app-specific functionality again needed by App B and .dll D. For instance, it includes an object factory. The app asks the factory to create objects with string names from a config file. The .dll needs to register the objects it has available with this factory, so also needs to access that factory and that's the specific part that MUST be a singleton. Instead the dll's objects register with THEIR factory, then app sees an empty one. — Swiss Frank, Oct 04 '22 at 12:01
I think the behavior is normal. There are 2 factories (one in the app and one in the *.dll*). If the logger would be itself in a separate *.dll*, it wouldn't matter as it would only be one instance per process. — CristiFati, Oct 04 '22 at 12:19
Not just the logger (lib B) but the functionality used by most code in this ecosystem (lib C). You are saying that if I make them both .so the problem will go away and I'll just have one singleton? Or is there any other options I can give the linker when linking my binary so that he can will allow lib D to load and link to the B and C in the app, instead of needing to be linked to its own copies? To be clear, even if it is "normal," my app doesn't work, and I need to fix it :-D I can't just tell my boss that it doesn't work but it's "normal." — Swiss Frank, Oct 04 '22 at 19:00
[\[SO\]: How to create a Minimal, Reproducible Example (reprex (mcve))](https://stackoverflow.com/help/minimal-reproducible-example). Try reproducing the problem with 3 simple source files. — CristiFati, Oct 05 '22 at 04:07
I'm surprised to hear you mention that only after you've shown you understand the problem perfectly. — Swiss Frank, Oct 05 '22 at 09:15
I'm not aware of a way to "trick" the linker to do what you want (but that doesn't mean it isn't). That's why I suggested to have a *MCVE*: one file with a (dummy) function relying on a static variable (to be the static (coincidence) lib), and 2 other (*.dll* and app) which use the static lib. It would be easier to reproduce the problem and attempt various flags to see if anything changes. It would also be easy to spot whether having a dynamic lib instead of the static one would solve your issue. — CristiFati, Oct 05 '22 at 09:33
B and C should be converted to shared objects to avoid code-duplication. — Lorinczy Zsigmond, Oct 06 '22 at 19:43
Yes, Loinczy, that was my ultimate solution. Christi's provided such a thorough answer that I don't want to supply a short version and mark it as correct, but if you write that as an answer I'd market it as correct. — Swiss Frank, Oct 08 '22 at 09:30
@EmployedRussian: Just out of curiosity: what would happen when generalizing? E.g. if there was another *.dll*: *libX* that was in the same situation as *libD* which was also loaded by the application? — CristiFati, Oct 09 '22 at 09:40

CristiFati · Answer 1 · 2022-10-09T10:00:40.170

Functionality from the static library exists independently in the:

.dll - it would work when loaded from another app
app - it works when not loading the .dll

So, I find the behavior you're experiencing normal (more: this would happen for each .dll containing the code from the static lib that is being loaded by the process).
Here's the MCVE ([SO]: How to create a Minimal, Reproducible Example (reprex (mcve))) that I was talking about (gVar (from lib00.c) could play the Singleton's RefCount role).

lib00.h:

#pragma once

#if defined(_WIN32)
#  if defined(LIB00_STATIC)
#    define LIB00_EXPORT_API
#  else
#    if defined(LIB00_EXPORTS)
#      define LIB00_EXPORT_API __declspec(dllexport)
#    else
#      define LIB00_EXPORT_API __declspec(dllimport)
#    endif
#  endif
#else
#  define LIB00_EXPORT_API
#endif


#if defined(__cplusplus)
extern "C" {
#endif

LIB00_EXPORT_API int libFunc();

#if defined(__cplusplus)
}
#endif

lib00.c:

#include <inttypes.h>
#include <stdio.h>

#define LIB00_EXPORTS
#include "lib00.h"

static int gVar = 0;

int libFunc()
{
    printf("  %s - var (0x%016lX): %d\n", __FUNCTION__, (uintptr_t)(&gVar), gVar);  // @TODO - cfati: technically, 1st arg yields UB (on Win)
    return gVar++;
}

dll00.c:

#include <stdio.h>

#include "lib00.h"

#if defined(_WIN32)
#  define DLL00_EXPORT_API __declspec(dllexport)
#else
#  define DLL00_EXPORT_API
#endif


#if defined(__cplusplus)
extern "C" {
#endif

DLL00_EXPORT_API void dllFunc();

#if defined(__cplusplus)
}
#endif


void dllFunc()
{
    printf("Call libFunc from .DLL: %d\n", libFunc());
}

main00.c:

#include <stdio.h>

#if defined (_WIN32)
#  include <windows.h>
#  define DLLPTR HMODULE
#  define dlsym GetProcAddress
#  define dlclose FreeLibrary
#else
#  include <dlfcn.h>
#  define DLLPTR void*
#endif

#include "lib00.h"


typedef int (*DllFuncPtr)();


int main(int argc, char **argv)
{
    DLLPTR pDll00 = NULL;
    DllFuncPtr pDllFunc = NULL;
    const int count = 3;
    if (argc > 1) {
#if defined (_WIN32)
        pDll00 = LoadLibrary(argv[1]);
#else
        int dlopen_flags = RTLD_NOW;  //RTLD_LAZY;
        //dlopen_flags |= RTLD_GLOBAL;
        pDll00 = dlopen(argv[1], dlopen_flags);
#endif
        if (pDll00) {
            pDllFunc = (DllFuncPtr)dlsym(pDll00, "dllFunc");
        } else {
            printf("Error loading .dll\n");
#if defined (_WIN32)
            printf("Error: %d\n", GetLastError());
#else
            printf("%s\n", dlerror());
#endif
        }
    }
    for (int i = 0; i < count; ++i) {
        if (pDllFunc)
            pDllFunc();
        printf("Call libFunc from .EXE: %d\n", libFunc());
    }
    if (pDll00)
        dlclose(pDll00);
    printf("\nDone.\n\n");
    return 0;
}

Don't mind the #defines, they are (mostly) for Win.

Output:

(qaic-env) [cfati@cfati-5510-0:/mnt/e/Work/Dev/StackOverflow/q073944078]> ~/sopr.sh
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[064bit prompt]> ls
dll00.c  lib00.c  lib00.h  main00.c
[064bit prompt]>
[064bit prompt]> # Build static lib00
[064bit prompt]>
[064bit prompt]> gcc -c -DLIB00_STATIC -o lib00s.o lib00.c
[064bit prompt]> ar rcs lib00s.a lib00s.o
[064bit prompt]> gcc -DLIB00_STATIC -fPIC -shared -o dll00s.so dll00.c lib00s.a
[064bit prompt]> gcc -DLIB00_STATIC -o app00s main00.c -ldl lib00s.a
[064bit prompt]> ls
app00s  dll00.c  dll00s.so  lib00.c  lib00.h  lib00s.a  lib00s.o  main00.c
[064bit prompt]>
[064bit prompt]> ./app00s
  libFunc - var (0x00005580D94BA014): 0
Call libFunc from .EXE: 0
  libFunc - var (0x00005580D94BA014): 1
Call libFunc from .EXE: 1
  libFunc - var (0x00005580D94BA014): 2
Call libFunc from .EXE: 2

Done.

[064bit prompt]> ./app00s ./dll00s.so
  libFunc - var (0x00007F4559918034): 0
Call libFunc from .DLL: 0
  libFunc - var (0x000056083959F014): 0
Call libFunc from .EXE: 0
  libFunc - var (0x00007F4559918034): 1
Call libFunc from .DLL: 1
  libFunc - var (0x000056083959F014): 1
Call libFunc from .EXE: 1
  libFunc - var (0x00007F4559918034): 2
Call libFunc from .DLL: 2
  libFunc - var (0x000056083959F014): 2
Call libFunc from .EXE: 2

Done.

[064bit prompt]>
[064bit prompt]> # Build dynamic lib00
[064bit prompt]>
[064bit prompt]> gcc -fPIC -shared -o lib00.so lib00.c
[064bit prompt]> gcc -fPIC -shared -o dll00.so dll00.c lib00.so
[064bit prompt]> gcc -o app00 main00.c -ldl lib00.so
[064bit prompt]> ls
app00  app00s  dll00.c  dll00.so  dll00s.so  lib00.c  lib00.h  lib00.so  lib00s.a  lib00s.o  main00.c
[064bit prompt]>
[064bit prompt]> ./app00
./app00: error while loading shared libraries: lib00.so: cannot open shared object file: No such file or directory
[064bit prompt]> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:. ./app00
  libFunc - var (0x00007FDCC6DC202C): 0
Call libFunc from .EXE: 0
  libFunc - var (0x00007FDCC6DC202C): 1
Call libFunc from .EXE: 1
  libFunc - var (0x00007FDCC6DC202C): 2
Call libFunc from .EXE: 2

Done.

[064bit prompt]> LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:. ./app00 ./dll00.so
  libFunc - var (0x00007FE562D1E02C): 0
Call libFunc from .DLL: 0
  libFunc - var (0x00007FE562D1E02C): 1
Call libFunc from .EXE: 1
  libFunc - var (0x00007FE562D1E02C): 2
Call libFunc from .DLL: 2
  libFunc - var (0x00007FE562D1E02C): 3
Call libFunc from .EXE: 3
  libFunc - var (0x00007FE562D1E02C): 4
Call libFunc from .DLL: 4
  libFunc - var (0x00007FE562D1E02C): 5
Call libFunc from .EXE: 5

Done.

So, moving functionality in a .dll solves the problem.
Having both builds, I wanted to see what happens when combining things:

[064bit prompt]> _LD_LIBRARY_PATH=${LD_LIBRARY_PATH}
[064bit prompt]> LD_LIBRARY_PATH=${_LD_LIBRARY_PATH}:.
[064bit prompt]> for g in app00s app00; do for i in dll00s.so dll00.so; do echo ./${g} ./${i}; ./${g} ./${i}; done done
./app00s ./dll00s.so
  libFunc - var (0x00007F5608ED4034): 0
Call libFunc from .DLL: 0
  libFunc - var (0x000055C0AFBCD014): 0
Call libFunc from .EXE: 0
  libFunc - var (0x00007F5608ED4034): 1
Call libFunc from .DLL: 1
  libFunc - var (0x000055C0AFBCD014): 1
Call libFunc from .EXE: 1
  libFunc - var (0x00007F5608ED4034): 2
Call libFunc from .DLL: 2
  libFunc - var (0x000055C0AFBCD014): 2
Call libFunc from .EXE: 2

Done.

./app00s ./dll00.so
  libFunc - var (0x00007FB3DBAB102C): 0
Call libFunc from .DLL: 0
  libFunc - var (0x000055BA0A2C5014): 0
Call libFunc from .EXE: 0
  libFunc - var (0x00007FB3DBAB102C): 1
Call libFunc from .DLL: 1
  libFunc - var (0x000055BA0A2C5014): 1
Call libFunc from .EXE: 1
  libFunc - var (0x00007FB3DBAB102C): 2
Call libFunc from .DLL: 2
  libFunc - var (0x000055BA0A2C5014): 2
Call libFunc from .EXE: 2

Done.

./app00 ./dll00s.so
  libFunc - var (0x00007F52A75A302C): 0
Call libFunc from .DLL: 0
  libFunc - var (0x00007F52A75A302C): 1
Call libFunc from .EXE: 1
  libFunc - var (0x00007F52A75A302C): 2
Call libFunc from .DLL: 2
  libFunc - var (0x00007F52A75A302C): 3
Call libFunc from .EXE: 3
  libFunc - var (0x00007F52A75A302C): 4
Call libFunc from .DLL: 4
  libFunc - var (0x00007F52A75A302C): 5
Call libFunc from .EXE: 5

Done.

./app00 ./dll00.so
  libFunc - var (0x00007FE3C8C9602C): 0
Call libFunc from .DLL: 0
  libFunc - var (0x00007FE3C8C9602C): 1
Call libFunc from .EXE: 1
  libFunc - var (0x00007FE3C8C9602C): 2
Call libFunc from .DLL: 2
  libFunc - var (0x00007FE3C8C9602C): 3
Call libFunc from .EXE: 3
  libFunc - var (0x00007FE3C8C9602C): 4
Call libFunc from .DLL: 4
  libFunc - var (0x00007FE3C8C9602C): 5
Call libFunc from .EXE: 5

Done.

For me, results are a bit odd, as I was expecting the 3^rd run to be identical to the previous 2. I must be missing something about symbol resolution (although the Update #0 section (at the end) sheds some light). Most likely, things could be altered by:

Changing dlopen flags
Modifying the static lib symbol (libFunc) visibility
Other ways (again, check Update #0 section at the end)

but most of them would require partial or full rebuild.

I did the same thing on Win:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q073944078]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2019\VC\Auxiliary\Build\vcvarsall.bat" x64 > nul

[prompt]> dir /b
app00
app00s
dll00.c
dll00.so
dll00s.so
lib00.c
lib00.h
lib00.so
lib00s.a
lib00s.o
main00.c

[prompt]>
[prompt]> cl -c /nologo /DLIB00_STATIC /MD lib00.c /Folib00s.obj
lib00.c
lib00.c(11): warning C4477: 'printf' : format string '%016lX' requires an argument of type 'unsigned long', but variadic argument 2 has type 'uintptr_t'
lib00.c(11): note: consider using '%llX' in the format string
lib00.c(11): note: consider using '%IX' in the format string
lib00.c(11): note: consider using '%I64X' in the format string

[prompt]> lib /NOLOGO /OUT:lib00s.lib lib00s.obj

[prompt]> cl /nologo /MD /DDLL lib00.c  /link /NOLOGO /DLL /OUT:lib00.dll
lib00.c
lib00.c(11): warning C4477: 'printf' : format string '%016lX' requires an argument of type 'unsigned long', but variadic argument 2 has type 'uintptr_t'
lib00.c(11): note: consider using '%llX' in the format string
lib00.c(11): note: consider using '%IX' in the format string
lib00.c(11): note: consider using '%I64X' in the format string
   Creating library lib00.lib and object lib00.exp

[prompt]>
[prompt]> cl /nologo /MD /DDLL /DLIB00_STATIC dll00.c  /link /NOLOGO /DLL /OUT:dll00s.dll lib00s.lib
dll00.c
   Creating library dll00s.lib and object dll00s.exp

[prompt]> cl /nologo /MD /DDLL dll00.c  /link /NOLOGO /DLL /OUT:dll00.dll lib00.lib
dll00.c
   Creating library dll00.lib and object dll00.exp

[prompt]>
[prompt]> cl /nologo /MD /W0 /DLIB00_STATIC main00.c  /link /NOLOGO /OUT:app00s.exe lib00s.lib
main00.c

[prompt]> cl /nologo /MD /W0 main00.c  /link /NOLOGO /OUT:app00.exe lib00.lib
main00.c

[prompt]>
[prompt]> dir /b
app00
app00.exe
app00s
app00s.exe
dll00.c
dll00.dll
dll00.exp
dll00.lib
dll00.obj
dll00.so
dll00s.dll
dll00s.exp
dll00s.lib
dll00s.so
lib00.c
lib00.dll
lib00.exp
lib00.h
lib00.lib
lib00.obj
lib00.so
lib00s.a
lib00s.lib
lib00s.o
lib00s.obj
main00.c
main00.obj

[prompt]>
[prompt]> for %g in (app00s.exe app00.exe) do (for %i in (dll00s.dll dll00.dll) do (echo %g %i && %g %i))

[prompt]> (for %i in (dll00s.dll dll00.dll) do (echo app00s.exe %i   && app00s.exe %i ) )

[prompt]> (echo app00s.exe dll00s.dll   && app00s.exe dll00s.dll )
app00s.exe dll00s.dll
  libFunc - var (0x000000009D343080): 0
Call libFunc from .DLL: 0
  libFunc - var (0x00000000DFD430D0): 0
Call libFunc from .EXE: 0
  libFunc - var (0x000000009D343080): 1
Call libFunc from .DLL: 1
  libFunc - var (0x00000000DFD430D0): 1
Call libFunc from .EXE: 1
  libFunc - var (0x000000009D343080): 2
Call libFunc from .DLL: 2
  libFunc - var (0x00000000DFD430D0): 2
Call libFunc from .EXE: 2

Done.


[prompt]> (echo app00s.exe dll00.dll   && app00s.exe dll00.dll )
app00s.exe dll00.dll
  libFunc - var (0x0000000099BE3060): 0
Call libFunc from .DLL: 0
  libFunc - var (0x00000000DFD430D0): 0
Call libFunc from .EXE: 0
  libFunc - var (0x0000000099BE3060): 1
Call libFunc from .DLL: 1
  libFunc - var (0x00000000DFD430D0): 1
Call libFunc from .EXE: 1
  libFunc - var (0x0000000099BE3060): 2
Call libFunc from .DLL: 2
  libFunc - var (0x00000000DFD430D0): 2
Call libFunc from .EXE: 2

Done.


[prompt]> (for %i in (dll00s.dll dll00.dll) do (echo app00.exe %i   && app00.exe %i ) )

[prompt]> (echo app00.exe dll00s.dll   && app00.exe dll00s.dll )
app00.exe dll00s.dll
  libFunc - var (0x0000000099BE3080): 0
Call libFunc from .DLL: 0
  libFunc - var (0x000000009D343060): 0
Call libFunc from .EXE: 0
  libFunc - var (0x0000000099BE3080): 1
Call libFunc from .DLL: 1
  libFunc - var (0x000000009D343060): 1
Call libFunc from .EXE: 1
  libFunc - var (0x0000000099BE3080): 2
Call libFunc from .DLL: 2
  libFunc - var (0x000000009D343060): 2
Call libFunc from .EXE: 2

Done.


[prompt]> (echo app00.exe dll00.dll   && app00.exe dll00.dll )
app00.exe dll00.dll
  libFunc - var (0x000000009D343060): 0
Call libFunc from .DLL: 0
  libFunc - var (0x000000009D343060): 1
Call libFunc from .EXE: 1
  libFunc - var (0x000000009D343060): 2
Call libFunc from .DLL: 2
  libFunc - var (0x000000009D343060): 3
Call libFunc from .EXE: 3
  libFunc - var (0x000000009D343060): 4
Call libFunc from .DLL: 4
  libFunc - var (0x000000009D343060): 5
Call libFunc from .EXE: 5

Done.

Here, things look as I would expect them to be.

AFAIC this is a PoC for .dll (.so) existence. Code (and data) resides in a (shared) location where it's accessed from by all its clients. No surprises.
An use-case that comes into my mind is upgrading (or modifying) the .lib:

static: recompile everything that links to it (and it might go further, recursively) - which is a nightmare
dynamic: simply replace the binary (assuming that it's backward compatible (API / ABI))

Update #0

[SO]: Multiple instances of singleton across shared libraries on Linux (@EmployedRussian's answer) (that this question was marked as a duplicate of) solves your problem too. I wonder how come I missed -rdynamic ([GNU.GCC]: Options for Linking, --export-dynamic ([SourceWare]: Command Line Options)), as I've used it in the past X( .

[064bit prompt]> gcc -DLIB00_STATIC -o app00srdyn main00.c -ldl -rdynamic lib00s.a
[064bit prompt]>
[064bit prompt]> ./app00srdyn ./dll00s.so
  libFunc - var (0x0000560504168014): 0
Call libFunc from .DLL: 0
  libFunc - var (0x0000560504168014): 1
Call libFunc from .EXE: 1
  libFunc - var (0x0000560504168014): 2
Call libFunc from .DLL: 2
  libFunc - var (0x0000560504168014): 3
Call libFunc from .EXE: 3
  libFunc - var (0x0000560504168014): 4
Call libFunc from .DLL: 4
  libFunc - var (0x0000560504168014): 5
Call libFunc from .EXE: 5

Done.

[064bit prompt]> ./app00srdyn ./dll00.so
  libFunc - var (0x000056162E6C2014): 0
Call libFunc from .DLL: 0
  libFunc - var (0x000056162E6C2014): 1
Call libFunc from .EXE: 1
  libFunc - var (0x000056162E6C2014): 2
Call libFunc from .DLL: 2
  libFunc - var (0x000056162E6C2014): 3
Call libFunc from .EXE: 3
  libFunc - var (0x000056162E6C2014): 4
Call libFunc from .DLL: 4
  libFunc - var (0x000056162E6C2014): 5
Call libFunc from .EXE: 5

Done.

Hello again @ChristiFati . I am now looking at your Windows version. If I understand correctly, app00.exe and dll00.dll are both linked with a DYNAMIC version of lib00, where as app00s.exe and dll00s.exe are both linked with a STATIC version, correct? — Swiss Frank, Nov 10 '22 at 20:20

dlopen() of a shared library results in multiple singletons

1 Answers1

Update #0