What is the expected relation of C++ modules and dynamic linkage?

Question

The C++ modules TS provides an excellent facility for eliminating the preprocessor, improving compile times, and generally supporting much more robust, modular, code development in C++, for non-template code at least.

The underlying machinery provides control over import and export of symbols in ordinary programs.

However, there is a major problem developing libraries for two kinds of dynamic loading: startup time loading, and run time loading. This problem involves the exporting of symbols from the library, which is often discussed in terms of visibility.

Generally, not all the extern symbols of the translation units used to construct a dynamic link library should be made visible to the user. In addition, with run time loading, especially with a plugin concept, the same symbol must be exported from many concurrently loaded libraries.

On Windows the use of language extensions

 __declspec(dllexport)
 __declspec(dllimport)

attached in source code as attributes of symbols, and more recently on gcc and clang systems on unix platforms, the use of

__attribute__((visibility("default")))
__attribute__((visibility("hidden"))) 

are intended to support the provision and use of symbols intended to be made public by the library. Using these is complicated and messy: on Windows macros must be used to export the symbols whilst compiling the library, but import them when using it. On the unix platforms, the visibility must be set to default to both export and import the symbols, the compiler deciding itself, based on whether a definition is found or not: the compiler must be invoked with

-fvisibility=hidden 

switch. The export/import attributes are not required for static linkage, and probably should be macro'd out to an empty string. Making code and fiddling the build system so that this all works, especially considering that #includes must have the correct symbol visibility set during compilation of library translation units is very hard, the file structure required in repositories is a mess, the source code is littered with macros, and in general .. the whole thing is a disaster. Almost all open source repositories FAIL to correctly export symbols for dynamic linkage, and most programmers have no idea that dynamic library code structure (using two level namespaces) is quite different to static linkage.

An example of how to do it (hopefully correctly) can be seen here:

https://github.com/calccrypto/uint256_t

This repository used to have 2 headers and 2 implementation files, the user of the built library would see 2 headers. There are now 7 headers and 2 implementation files and the user of the built library will see 5 header files (3 with extension include to indicate they're not to be directly included).

So after that long winded explanation, the question is: will the final C++ modules specification help to solve problems with export and import of symbols for dynamic linkage? Can we expect to be able to develop for shared libraries without polluting our code with vendor specific extensions and macros?

Answer 1

Modules don't help you with symbol visibility across DLL boundaries. We can check this with a quick experiment.

// A.ixx
export module A;

export int f() { return 1; }

Here we have a simple module interface file exporting one symbol f in the module interface of module A (happens to share the same name as the file base name, but this isn't necessary). Let's compile this like so:

cl /c /std:c++20 /interface /TP A.ixx

The /c flag avoids invoking the linker (happens automatically by default), c++20 or later is required for module syntax to work, and the /interface flag lets the compiler know we are compiling a module interface unit. The /TP arg says "treat the source input as a C++ input" and is needed when /interface is specified. Finally, we have our input file.

Running the above produces an interface file A.ifc and an object file A.obj. Note that there is no import lib file or exp file you would expect if you were compiling a DLL.

Next, let's write a file that consumes this.

// main.cpp
import A;

int main() { return f(); }

To compile this into an executable, we can use the following command:

cl /std:c++20 main.cpp A.obj

The presence of the A.obj input there is not optional. Without it, we get a classic linker error of f being an unresolved symbol. If we run this, we'll get a main.exe which statically links the code in A.obj.

What happens if we try and compile A.ixx into a DLL? That is, what if we try to produce a DLL with the linker from A.obj? The answer you get a DLL but no import lib or exp. If you try running link /noentry /dll A.obj /out:A.dll you will get an A.dll with the expected /disasm section (visible via dump bin), but no export table.

Dump of file A.dll

File Type: DLL

  0000000180001000: B8 01 00 00 00     mov         eax,1
  0000000180001005: C3                 ret

  Summary

        1000 .rdata
        1000 .text

That's the disassembly in A.dll which we expect, but checking the exports section with dumpbin /export A.dll reveals nothing. The reason is of course, we didn't export the symbol!

If we change the source code of A.ixx to the following:

// A.ixx
export module A;

export __declspec(export) int f() { return 1; }

... we can repeat the steps (compile A.obj, link A.dll) to find that this time, the linker produces an import lib and exp file as we'd expect. Invoking dumpbin /exports A.lib on the import lib generated should show the ?f@@YAHXZ symbol present.

Now, we can link main.cpp again A.lib (as opposed to A.obj) via cl /std:c++20 main.cpp A.lib to produce a valid executable, this time relying on A.dll for the code instead of having f statically embedded.

We can check that this is in fact happening as expected in WinDbg.

Note on the lower left module pane the presence of A.dll. Note also that in the disassembly view in the center, we are about to call main!f. Uh oh, not good. While this does properly resolve to the !A module, it does so via an extra indirection in the import address table as seen here:

This is the classic problem that happens when you forget to decorate a function or symbol with the __declspec(dllimport) directive. When the compiler encounters the symbol without the dllimport directive that it doesn't recognize, it emits a relocation entry which is expected to be resolved at link time. Along with that entry, it emits a jmp and an unresolved address. This is a classic problem that I won't get into here, but the upshot is that we have an extra unnecessary indirection because the symbol recognized as exported from the module A was expected to be statically linked.

It turns out, we can't fix this easily. If we try to add another declaration of f to main.cpp or some other translation unit, the linker will complain that it sees f with "inconsistent dll linkage." The only way to resolve this is to compile a second version of the A module interface with dllimport decorations (much like how headers typically have macros that expand to dllexport or dllimport depending on the TU using the header).

The moral of the story is that DLL linkage and module linkage, while not completely at odds, aren't particularly compatible either. The module export does not include exported symbols in the export table, needed to resolve symbols across DLL boundaries. Furthermore, putting these symbols in the export table still leaves you the trouble of an extra indirection after the implicit dynamic link is done via the import address table.