CMake: how create a single shared library from all static libraries of subprojects?

Question

I have the following layout:

top_project
    + subproject1
    + subproject2

Each of subproject1 and subproject2 creates a static library. I would like to link these static libraries in a single shared library at the top_project level.

The information I gathered so far is:

Either compile using -fPic (necessary on everything but Windows) in order to create position-independent code which will allow linking the static libraries into a single shared library or decompress all static libraries (e.g. using ar) and re-link them into a shared library (which I think is an inelegant & non-portable solution)
All source files must be given explicitly to the add_library command: for some reason which I cannot comprehend, simply writing add_library(${PROJECT_NAME} SHARED subproject1 subproject2) does not work as expected (it essentially creates an empty library & does not register the dependencies properly)
There is an OBJECT library feature in CMake but I don't think it's purpose is really to do what I want.

Any thoughts?

I'm using cmake 3.4.+ and i simply add static libs to shared lib and they are compiled to single file :) I tested this on android :) — Gelldur, Nov 16 '16 at 10:54
would anyone have a hint on how to do this under MSVC? I am using qmake and not cmake but I can handle the steps myself, if I can figure them out... — S. Paris, May 04 '17 at 13:50

score 44 · Answer 1 · edited Sep 09 '16 at 17:24

OK, I figured it out: this is much more painful than it should be. Until very recently, people at Kitware didn't understand why anyone would ever want to create a DLL from static libs. Their argument is that there should always be source files in the main (e.g. top_project in my case) directory because it is effectively a project of its own. I see things differently & I need to break top_project into smaller subprojects which should not exist independently (i.e. there is no point in creating a full-blown project for them & add them using ExternalProject_Add). Besides, when I ship my shared library (for use, e.g. with a Java Native Interface), I don't want to ship dozens of shared libraries because that would amount to exposing the internal layout of my project. Anyway, having - I think - made a case for creating a shared library from static libraries, I'll proceed to the technical details.

In the CMakeLists.txt of subproject1 and subproject2, you should create your target using the OBJECT library feature (introduced in CMake 2.8.8):

add_library(${PROJECT_NAME} OBJECT ${SRC})

where SRC designates the list of source files (note that these should be set explicitly in the CMakeLists.txt file as it allows make to re-launch CMake when a modification of CMakeLists.txt is detected, e.g. when adding or removing a file)

In the top_project, add the subprojects using:

add_subdirectory(subproject1)
add_subdirectory(subproject2)

In order to see the symbols from the static library, use:

set(CMAKE_SHARED_LINKER_FLAGS "-Wl,--export-all-symbols")

You can then create the shared library using:

add_library(${PROJECT_NAME} SHARED $<TARGET_OBJECTS:subproject1>
                                   $<TARGET_OBJECTS:subproject2>)

I've found that any "normal" library (i.e. not object) needs to be added in a separate add_library command, otherwise it is simply ignored.

For executables, you can use:

add_executable(name_of_executable $<TARGET_OBJECTS:subproject1>
                  $<TARGET_OBJECTS:subproject2>)
set(LINK_FLAGS ${LINK_FLAGS} "-Wl,-whole-archive")
target_link_libraries(name_of_executable ${PROJECT_NAME}

I repeat that this only works as of version 2.8.8 of CMake. Just as well CMake manages the dependencies extremely well & is cross-platform because it's not much less painful than plain old Makefiles & certainly less flexible.

Bah, annoyingly Ubuntu 12.04 is stuck on CMake 2.8.7, is there an alternative for older versions? Do we just have to refer to all the source files when defining the library? — Ibrahim, Nov 21 '12 at 05:52
I worked around my issues by compiling my static libraries with -fPIC, my shared library did link properly but I dunno if it actually works though since I haven't tried using it yet. — Ibrahim, Nov 21 '12 at 07:34
Note: This works even if the source libraries subproject1, subproject2 are STATIC and not OBJECT. $ is listing all of the objects required to build the sub project, and does not pay attention to what the original library being built was. This could also work for sub projects that build SHARED libraries as well. Thank you very much for this answer. CMake documentation tells you there is a thing, but never gives you much context. Well, it has been getting better. — natersoz, Dec 13 '22 at 17:20

Timmmm · Answer 2 · 2019-07-15T09:36:37.267

My solution is simply to add /WHOLEARCHIVE, -all_load, or --whole-archive to the linker flags, so that when your main library is linked, all of the sub libraries are included, including all their symbols (the default behaviour is to only include symbols of the sub libraries that are used by the main library. For example:

Source Files

$ echo "void Func1() { }" > source1.cpp
$ echo "void Func2() { }" > source2.cpp
$ echo "void Func3() { }" > source3.cpp
$ echo "void Func4() { }" > source4.cpp

Naive CMakeLists.txt

cmake_minimum_required(VERSION 3.7)

# The 'sub' libraries, e.g. from an `add_subdirectory()` call.
add_library(sublib_a STATIC source1.cpp source2.cpp)
add_library(sublib_b STATIC source3.cpp source4.cpp)

# The main library that contains all of the sub libraries.
add_library(mainlib SHARED)

target_link_libraries(mainlib sublib_a sublib_b)

Running it (on OSX):

$ make VERBOSE=1
...
[100%] Linking CXX shared library libmainlib.dylib
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E cmake_link_script CMakeFiles/mainlib.dir/link.txt --verbose=1
/Library/Developer/CommandLineTools/usr/bin/c++   -dynamiclib -Wl,-headerpad_max_install_names  -o libmainlib.dylib -install_name @rpath/libmainlib.dylib  libsublib_a.a libsublib_b.a 
[100%] Built target mainlib

$ nm libmainlib.dylib | grep Func
$

Correct CMakeLists.txt

Append this:

# By default, symbols provided by the sublibs that are not used by mainlib (which is all of them in this case)
# are not used. This changes that.
if (WIN32)
    set_target_properties(mainlib PROPERTIES
        LINK_FLAGS "/WHOLEARCHIVE"
    )
elseif (APPLE)
    set_target_properties(mainlib PROPERTIES
        LINK_FLAGS "-Wl,-all_load"
    )
else ()
    set_target_properties(mainlib PROPERTIES
        LINK_FLAGS "-Wl,--whole-archive"
    )
endif ()

Running it (note the extra -all_load):

$ make VERBOSE=1
[100%] Linking CXX shared library libmainlib.dylib
/usr/local/Cellar/cmake/3.7.1/bin/cmake -E cmake_link_script CMakeFiles/mainlib.dir/link.txt --verbose=1
/Library/Developer/CommandLineTools/usr/bin/c++   -dynamiclib -Wl,-headerpad_max_install_names -Wl,-all_load -o libmainlib.dylib -install_name @rpath/libmainlib.dylib  libsublib_a.a libsublib_b.a 
[100%] Built target mainlib

$ nm libmainlib.dylib | grep Func
0000000000001da0 T __Z5Func1v
0000000000001db0 T __Z5Func2v
0000000000001dc0 T __Z5Func3v
0000000000001dd0 T __Z5Func4v

Note that I've only actually tested -all_load so far, and /WHOLEARCHIVE is an MSVC 2015 option.

I'm on Linux, simply adding `-Wl,--whole-archive` resulted in tons of "multiple definitions" errors related to `libgcc.a` — nodakai, Mar 19 '17 at 23:02
Yeah the `/WHOLEARCHIVE` option doesn't seem to work that well either so I just went with the object library approach. — Timmmm, Mar 20 '17 at 15:42
I really wanted this approach to work, but it is not robust enough. Even if you deal with the whole archive flag correctly... if two libraries have an identical symbol name it will fail with multiple definition errors. Object library is probably the way to go, but it typically requires a significant overhaul to the CMake projects you are trying to combine. — CraigDavid, Apr 01 '23 at 00:16

score 4 · Answer 3 · edited May 23 '17 at 12:02

4

Another way of doing it.

This way seems simpler, but I'm not sure how perfect it is:

https://stackoverflow.com/a/14347487/602340

edited May 23 '17 at 12:02

Community

1
1

answered Jan 15 '13 at 21:44

matiu

7,469
4
44
48

score 2 · Answer 4 · answered May 29 '15 at 20:07

Another way of doing it is to provide the path of the source files and the header files of all your projects, and build them together to produce the .so . This is usually the recommended way, instead of creating the static libraries and then a shared library out of those.

Basically you should do the following:

FILE(GLOB subproject1_sources
  <sub_project1_lib_sources_dir>/file1.c
  <sub_project1_lib_sources_dir>/file2.c //... etc
)

FILE(GLOB subproject2_sources
  <sub_project2_lib_sources_dir>/file1.c
  <sub_project2_lib_sources_dir>/file2.c //... etc
)

FILE(GLOB topProject_sources
  <top_project_lib_sources_dir>/file1.c
  <top_project_lib_sources_dir>/file2.c //... etc
)

include_directories("<sub_project1_lib_sources_dir>")
include_directories("<sub_project2_lib_sources_dir>")
include_directories("<top_project_lib_sources_dir>") //should be "." if you're building from here

add_library(topProject SHARED ${topProject_sources} ${subproject1_sources} ${subproject2_sources})

This is very unlikely to be useful. The problem is that you often have a mix of older build systems that produce .o files by some internal magic that you DO NOT want to change. You're rarely going to just be able to add a list of sources in any production system. — James Moore, Jan 05 '18 at 00:23

score 1 · Answer 5 · answered May 01 '20 at 19:29

I am not sure if this is what suits your need, but cmake also offers INTERFACE libraries, which serve (among others) precisely this need.

add_library(bundle INTERFACE)
target_link_libraries(bundle lib1 lib2)

will bundle lib1 and lib2 into a single library, and inherit the PUBLIC and INTERFACE section of lib1 and lib2.

More info here.

score 0 · Answer 6 · answered Aug 27 '21 at 12:15

Add following macro to your cmake scripts.

MACRO (TARGET_LINK_LIBRARIES_WHOLE_ARCHIVE target)
  IF (WIN32)
    FOREACH (arg IN LISTS ARGN)
      SET_TARGET_PROPERTIES(
        ${target} PROPERTIES LINK_FLAGS "/WHOLEARCHIVE:${lib}"
      )
    ENDFOREACH ()
  ELSE ()
    IF (APPLE)
      SET(LINK_FLAGS "-Wl,-all_load")
      SET(UNDO_FLAGS "-Wl,-noall_load")
    ELSE ()
      SET(LINK_FLAGS "-Wl,--whole-archive")
      SET(UNDO_FLAGS "-Wl,--no-whole-archive")
    ENDIF ()
    TARGET_LINK_LIBRARIES(${target} ${LINK_FLAGS} ${ARGN} ${UNDO_FLAGS})
  ENDIF ()
ENDMACRO ()

Then, in CMakeLists.txt/*.cmake, you can use like this TARGET_LINK_LIBRARIES_WHOLE_ARCHIVE(target libs...)

CMake: how create a single shared library from all static libraries of subprojects?

6 Answers6

Source Files

Naive CMakeLists.txt

Correct CMakeLists.txt

Linked