Combining several static libraries into one using CMake

Question

I have a very similar problem to one described on the cmake mailing list where we have a project dependent on many static libraries (all built from source in individual submodules, each with their own CMakeLists.txt describing the build process for each library) that I'd like to combine into a single static library for release to the consumers. The dependencies of my library are subject to change, and I do not want to burden developers further down the chain with those changes. The neat solution would be to bundle all of the libs into one single lib.

Interestingly, the target_link_libraries command does not combine all of the statics when setting the target to mylib and using it like so . .

target_link_libraries(mylib a b c d)

However, bizarrely, if I make the mylib project a submodule of an executable project, and only link against mylib in the top level executable CMAkeLists.txt, the library does seem to be combined. I.e. mylib is 27 MB, instead of the 3MB when I set the target to only build mylib.

There are solutions describing unpacking of the libs into object files and recombining (here, and here), but this seems remarkably clumsy when CMake seems perfectly capable of automatically merging the libs as described in the above example. It there a magic command I'm missing, or a recommended elegant way of making a release library?

@Drop they are all pimpled off or hidden behind the public interface of `mylib`. The deps should be invisible to the consumers — learnvst, Jun 20 '16 at 14:01
If you are using gcc, and don't require your makefile to be compiler independent, you could try the [`--whole-archive`](http://stackoverflow.com/questions/805555/ld-linker-question-the-whole-archive-option) option. — Karsten Koop, Jun 20 '16 at 14:11
Possible duplicate http://stackoverflow.com/questions/3821916/how-to-merge-two-ar-static-libraries-into-one — n. m. could be an AI, Jun 21 '16 at 10:27
Hmmm @n.m. I was hoping to have CMake do this in a platform independent way seeing as that is the whole point of the tool. My answer below works, but sucks because of its platform dependence — learnvst, Jun 21 '16 at 10:29
I think you want to study [this](http://www.mail-archive.com/cmake@cmake.org/msg28670/libutils.cmake). It's a fragment of mysql CMake build. They have their own macro to merge libraries. As you can see, when it comes to merging static libraries into a bigger static library, they branch into unix/apple/windows paths. I believe there's no truly portable way to do this. — n. m. could be an AI, Jun 21 '16 at 10:50
"the library does seem to be combined. I.e. mylib is 27 MB, instead of the 3MB" Try a verbose build and see what command is used to build the big library. — n. m. could be an AI, Jun 21 '16 at 10:53

learnvst · Answer 1 · 2016-06-21T10:19:09.710

Given the most simple working example I can think of: 2 classes, a and b, where a depends on b . .

a.h

#ifndef A_H
#define A_H

class aclass
{
public:
    int method(int x, int y);
};

#endif

a.cpp

#include "a.h"
#include "b.h"

int aclass::method(int x, int y) {
    bclass b;
    return x * b.method(x,y);
}

b.h

#ifndef B_H
#define B_H

class bclass
{
public:
    int method(int x, int y);
};

#endif

b.cpp

#include "b.h"

int bclass::method(int x, int y) {
    return x+y;
}

main.cpp

#include "a.h"
#include <iostream>

int main()
{
    aclass a;
    std::cout << a.method(3,4) << std::endl;

    return 0;
}

It is possible to compile these into separate static libs, and then combine the static libs using a custom target.

cmake_minimum_required(VERSION 2.8.7)

add_library(b b.cpp b.h)
add_library(a a.cpp a.h)
add_executable(main main.cpp)

set(C_LIB ${CMAKE_BINARY_DIR}/libcombi.a)

add_custom_target(combined
        COMMAND ar -x $<TARGET_FILE:a>
        COMMAND ar -x $<TARGET_FILE:b>
        COMMAND ar -qcs ${C_LIB} *.o
        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
        DEPENDS a b
        )

add_library(c STATIC IMPORTED GLOBAL)
add_dependencies(c combined)

set_target_properties(c
        PROPERTIES
        IMPORTED_LOCATION ${C_LIB}
        )

target_link_libraries(main c)

It also works just fine using Apple's libtool version of the custom target . . .

add_custom_target(combined
        COMMAND libtool -static -o ${C_LIB} $<TARGET_FILE:a> $<TARGET_FILE:b>
        WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
        DEPENDS a b
        )

Still seams as though there should be a neater way . .

This doesn't answer the question. This reply is about combining several object files while the question is about combining several libraries. — ctc chen, Nov 30 '16 at 08:57
@ctcchen: Well, it looks like the `CMakeLists.txt` will result in a combined library, `libcombi.a`. But my question to learnvst is: Does this really require a custom target? Isn't there a more standard mechanism? — einpoklum, Apr 02 '17 at 12:26
@ctcchen Nope, he creates 2 libraries "a" and "b" from a.cpp and b.cpp, he then combines them into library "c". — Jimmy Pettersson, Sep 20 '18 at 14:02
Thanks! the corresponding way to do it on Windows is to add a custom command "`lib.exe /OUT:combi.lib a.lib b.lib`" (good to know that CMake does not provide any helper method, and that we have to manually support each platform separately) — Top-Master, Jul 11 '21 at 19:30

score 12 · Answer 2 · answered Nov 08 '18 at 11:14

You can use this function to join any number of libraries.

function(combine_archives output_archive list_of_input_archives)
    set(mri_file ${TEMP_DIR}/${output_archive}.mri)
    set(FULL_OUTPUT_PATH ${CMAKE_ARCHIVE_OUTPUT_DIRECTORY}/lib${output_archive}.a)
    file(WRITE ${mri_file} "create ${FULL_OUTPUT_PATH}\n")
    FOREACH(in_archive ${list_of_input_archives})
        file(APPEND ${mri_file} "addlib ${CMAKE_ARCHIVE_OUTPUT_DIRECTORY}/lib${in_archive}.a\n")
    ENDFOREACH()
    file(APPEND ${mri_file} "save\n")
    file(APPEND ${mri_file} "end\n")

    set(output_archive_dummy_file ${TEMP_DIR}/${output_archive}.dummy.cpp)
    add_custom_command(OUTPUT ${output_archive_dummy_file}
                       COMMAND touch ${output_archive_dummy_file}
                       DEPENDS ${list_of_input_archives})

    add_library(${output_archive} STATIC ${output_archive_dummy_file})
    add_custom_command(TARGET ${output_archive}
                       POST_BUILD
                       COMMAND ar -M < ${mri_file})
endfunction(combine_archives)

It has the benefits of using add_custom_command and not add_custom_target. This way, the library (and it's dependencies) are only built when needed and not every time. The drawback is the print of the generation of the dummy file.

While this is good for combining the object files, I think it could be improved by adding a `target_link_libraries` call to get transitive dependencies. — Daniel Jour, Sep 24 '19 at 07:56
doesn't work on macos. The `-M` doesn't exist in the `ar` command on macos :( — Adham Zahran, Aug 30 '20 at 20:38
This should be the accepted answer, thanks! Can easily modify the above to use `libtool` for MacOS, allowing it to work on all Unix-like OS. — cyrusbehr, Aug 27 '21 at 17:37

driedler · Answer 3 · 2023-04-17T20:55:58.427

This doesn't directly answer the question, but I found it useful:

https://cristianadam.eu/20190501/bundling-together-static-libraries-with-cmake/

Basic, define a CMake function that will collect all the static libs required by a target and combine them into a single static lib:

add_library(awesome_lib STATIC ...);
bundle_static_library(awesome_lib awesome_lib_bundled)

Here's a copy & paste of the actual function:

function(bundle_static_library tgt_name bundled_tgt_name)
  list(APPEND static_libs ${tgt_name})

  function(_recursively_collect_dependencies input_target)
    set(_input_link_libraries LINK_LIBRARIES)
    get_target_property(_input_type ${input_target} TYPE)
    if (${_input_type} STREQUAL "INTERFACE_LIBRARY")
      set(_input_link_libraries INTERFACE_LINK_LIBRARIES)
    endif()
    get_target_property(public_dependencies ${input_target} ${_input_link_libraries})
    foreach(dependency IN LISTS public_dependencies)
      if(TARGET ${dependency})
        get_target_property(alias ${dependency} ALIASED_TARGET)
        if (TARGET ${alias})
          set(dependency ${alias})
        endif()
        get_target_property(_type ${dependency} TYPE)
        if (${_type} STREQUAL "STATIC_LIBRARY")
          list(APPEND static_libs ${dependency})
        endif()

        get_property(library_already_added
          GLOBAL PROPERTY _${tgt_name}_static_bundle_${dependency})
        if (NOT library_already_added)
          set_property(GLOBAL PROPERTY _${tgt_name}_static_bundle_${dependency} ON)
          _recursively_collect_dependencies(${dependency})
        endif()
      endif()
    endforeach()
    set(static_libs ${static_libs} PARENT_SCOPE)
  endfunction()

  _recursively_collect_dependencies(${tgt_name})

  list(REMOVE_DUPLICATES static_libs)

  set(bundled_tgt_full_name 
    ${CMAKE_BINARY_DIR}/${CMAKE_STATIC_LIBRARY_PREFIX}${bundled_tgt_name}${CMAKE_STATIC_LIBRARY_SUFFIX})

  if (CMAKE_CXX_COMPILER_ID MATCHES "^(Clang|GNU)$")
    file(WRITE ${CMAKE_BINARY_DIR}/${bundled_tgt_name}.ar.in
      "CREATE ${bundled_tgt_full_name}\n" )
        
    foreach(tgt IN LISTS static_libs)
      file(APPEND ${CMAKE_BINARY_DIR}/${bundled_tgt_name}.ar.in
        "ADDLIB $<TARGET_FILE:${tgt}>\n")
    endforeach()
    
    file(APPEND ${CMAKE_BINARY_DIR}/${bundled_tgt_name}.ar.in "SAVE\n")
    file(APPEND ${CMAKE_BINARY_DIR}/${bundled_tgt_name}.ar.in "END\n")

    file(GENERATE
      OUTPUT ${CMAKE_BINARY_DIR}/${bundled_tgt_name}.ar
      INPUT ${CMAKE_BINARY_DIR}/${bundled_tgt_name}.ar.in)

    set(ar_tool ${CMAKE_AR})
    if (CMAKE_INTERPROCEDURAL_OPTIMIZATION)
      set(ar_tool ${CMAKE_CXX_COMPILER_AR})
    endif()

    add_custom_command(
      COMMAND ${ar_tool} -M < ${CMAKE_BINARY_DIR}/${bundled_tgt_name}.ar
      DEPENDS ${static_libs}  
      OUTPUT ${bundled_tgt_full_name}
      COMMENT "Bundling ${bundled_tgt_name}"
      VERBATIM)
  elseif(MSVC)
    find_program(lib_tool lib)

    foreach(tgt IN LISTS static_libs)
      list(APPEND static_libs_full_names $<TARGET_FILE:${tgt}>)
    endforeach()

    add_custom_command(
      COMMAND ${lib_tool} /NOLOGO /OUT:${bundled_tgt_full_name} ${static_libs_full_names}
      DEPENDS ${static_libs}          
      OUTPUT ${bundled_tgt_full_name}
      COMMENT "Bundling ${bundled_tgt_name}"
      VERBATIM)
  else()
    message(FATAL_ERROR "Unknown bundle scenario!")
  endif()

  add_custom_target(bundling_target ALL DEPENDS ${bundled_tgt_full_name})
  add_dependencies(bundling_target ${tgt_name})

  add_library(${bundled_tgt_name} STATIC IMPORTED)
  set_target_properties(${bundled_tgt_name} 
    PROPERTIES 
      IMPORTED_LOCATION ${bundled_tgt_full_name}
      INTERFACE_INCLUDE_DIRECTORIES $<TARGET_PROPERTY:${tgt_name},INTERFACE_INCLUDE_DIRECTORIES>)
  add_dependencies(${bundled_tgt_name} bundling_target)

endfunction()

Thanks for sharing this! A small improvement to make sure that the target library is generated when one of the dependencies have changed: Add this line to the 'add_custom_command' lines: `DEPENDS ${static_libs}` — fredvanl, Mar 24 '22 at 06:29

score 4 · Answer 4 · answered Jun 14 '18 at 16:10

If the libraries you are trying to merge are from third parties, then (following learnvst example) this code take care of possible .o file replacements (if for instance both liba and libb have a file name zzz.o)

## Create static library (by joining the new objects and the dependencies)
ADD_LIBRARY("${PROJECT_NAME}-static" STATIC ${SOURCES})
add_custom_command(OUTPUT lib${PROJECT_NAME}.a
                   COMMAND rm ARGS -f *.o
                   COMMAND ar ARGS -x ${CMAKE_BINARY_DIR}/lib${PROJECT_NAME}-static.a
                   COMMAND rename ARGS 's/^/lib${PROJECT_NAME}-static./g' *.o
                   COMMAND rename ARGS 's/\\.o/.otmp/g' *.o
                   COMMAND ar ARGS -x ${CMAKE_SOURCE_DIR}/lib/a/liba.a
                   COMMAND rename ARGS 's/^/liba./g' *.o
                   COMMAND rename ARGS 's/\\.o/.otmp/g' *.o
                   COMMAND ar ARGS -x ${CMAKE_SOURCE_DIR}/lib/b/libb.a
                   COMMAND rename ARGS 's/^/libb./g' *.o
                   COMMAND rename ARGS 's/\\.o/.otmp/g' *.o
                   COMMAND rename ARGS 's/\\.otmp/.o/g' *.otmp
                   COMMAND ar ARGS -r lib${PROJECT_NAME}.a *.o
                   COMMAND rm ARGS -f *.o
                   DEPENDS "${PROJECT_NAME}-static")

add_custom_target(${PROJECT_NAME} ALL DEPENDS lib${PROJECT_NAME}.a)

Otherwise, if the libraries are yours, you should use CMake OBJECT libraries, that are a pretty good mechanism to get them merged.

You should almost certainly use `PROJECT_BINARY_DIR` and `PROJECT_SOURCE_DIR` instead of the `CMAKE_...` variants. It's getting increasingly popular to import other CMake projects into a "superbuild" and the answer above breaks in that scenario. — Alex Reinking, Apr 14 '20 at 22:18

score 1 · Answer 5 · answered Aug 17 '21 at 07:47

1

https://cmake.org/pipermail/cmake/2018-September/068263.html

It seems CMake doesn't support that.

answered Aug 17 '21 at 07:47

Ox9A82

29
5

score 1 · Answer 6 · answered Aug 17 '21 at 07:55

The proper way to do this is not to fudge around with combining static libraries, but to provide CMake Config files to the user that contain all the necessary bits that link everything the way it's supposed to be linked. CMake can be used to generate these files, or generate pkg-config files, and probably other formats of "tell me how to link with this and that library" tools.

Chances are some users will be interested in what libraries yours link to, and they might even be using their own copies/versions of the same exact libraries when linking yours. It is in exactly this case that your solution is terrible and prevents users from integrating multiple pieces of code, because you decided they must absolutely use your copy of that dependency (which is what you do when you combine static library dependencies into one static library).

Unless the library is intending to be used in projects not based on CMake. — Dávid Tóth, Sep 27 '22 at 18:55
...or the stuff is interleaved static libraries where something refers to something in the guts of say, newlib, but is used in an application as another .a ld won't let you do that- you have to aggregate the .a files or similar to get there. There is no tidy CMake-ish way to DO that one... Next time check to see what they're doing before mouthing off... — Svartalf, Apr 19 '23 at 19:55

score 1 · Answer 7 · answered Sep 27 '22 at 23:53

I created a solution based on zbut's answer, but it supports retrieving the input library paths from given targets and also supports multi-configuration generators like Ninja Multi-Config:

# Combine a list of library targets into a single output archive
# Usage:
# combine_archives(output_archive_name input_target1 input_target2...)
function(combine_archives output_archive)
    # Generate the MRI file for ar to consume.
    # Note that a separate file must be generated for each build configuration.
    set(mri_file ${CMAKE_BINARY_DIR}/$<CONFIG>/${output_archive}.mri)
    set(mri_file_content "create ${CMAKE_BINARY_DIR}/$<CONFIG>/lib${output_archive}.a\n")
    FOREACH(in_target ${ARGN})
        string(APPEND mri_file_content "addlib $<TARGET_FILE:${in_target}>\n")
    ENDFOREACH()
    string(APPEND mri_file_content "save\n")
    string(APPEND mri_file_content "end\n")
    file(GENERATE
            OUTPUT ${mri_file}
            CONTENT ${mri_file_content}
            )

    # Create a dummy file for the combined library
    # This dummy file depends on all the input targets so that the combined library is regenerated if any of them changes.
    set(output_archive_dummy_file ${CMAKE_BINARY_DIR}/${output_archive}.dummy.cpp)
    add_custom_command(OUTPUT ${output_archive_dummy_file}
            COMMAND touch ${output_archive_dummy_file}
            DEPENDS ${ARGN})

    add_library(${output_archive} STATIC ${output_archive_dummy_file})

    # Add a custom command to combine the archives after the static library is "built".
    add_custom_command(TARGET ${output_archive}
            POST_BUILD
            COMMAND ar -M < ${mri_file}
            COMMENT "Combining static libraries for ${output_archive}"
            )
endfunction(combine_archives)

Usage to generate a libTargetC.a from libTargetA.a and libTargetB.a would be something like:

add_library(TargetA STATIC ...)
add_library(TargetB STATIC ...)
combine_archives(TargetC TargetA TargetB)

Combining several static libraries into one using CMake

7 Answers7

a.h

a.cpp

b.h

b.cpp

main.cpp

Linked

Related