10

I need to figure out a compiler/linker directive I can feed into gcc so that it won't automatically link libgomp when -fopenmp is specified.

The reason is that I'm trying to build against Intel's MKL BLAS. MKL requires adding a separate Intel library to handle multithreading (e.g., libmkl_intel_thread or libmkl_gnu_thread). The library for linking MKL against libgomp, however, is not available on every operating system, including mine. This forces me to link libmkl_intel_thread, which in turn must link against libiomp5.

While I am able to build my package, some binaries are linked against both libgomp and libiomp5. I'm not positive this is causing problems, but there have been a few crashes, the linkage-combination is suspicious, and even if it isn't causing crashes its certainly a terrible inefficiency.

I'm trying to do this with gcc 4.9.1.

Avoiding -fopenmp is, unfortunately, not an option. The reason is that this is for compiling a rather large package comprised of several sub-packages, whose Makefiles are not in the greatest shape, and where additional packages from other sources (plug-ins) may be compiled later. Forcing a universal compiler/linker directive isn't difficult. Turning on --enable-openmp, however, activates both -fopenmp, and defines that are used to trigger code related to multi-threading. Trying to separate the three (--enable-openmp, -fopenmp, and code linked to --enable-openmp) isn't feasible.

I've looked through manual pages, and I don't see any directive for gcc that would allow selection of an openmp library. Intel's forums have a very old discussion in which they suggest specifying a static library immediately after -fopenmp followed by --as-needed. Which seems pretty rickety, and also has a lot of potential to interfere with plugin packages. llvm-openmp seems to have considered a -fopenmp=libiomp5 directive at one point, but it seems to have been dropped in the 3.5 release and I'm trying to use gcc anyway.

Thanks.

Bob
  • 1,274
  • 1
  • 13
  • 26

3 Answers3

8

GCC does not support linking against the Intel OpenMP runtime library. GCC's internal code transformer translates OpenMP directives into ligomp-specific calls and those have a way different API than the one exposed by libiomp. Also, mixing two separate OpenMP runtimes into one executable (or into a single process, if OpenMP-enabled modules are being loaded dynamically) is a recipe for disaster. That's the reason why MKL's multithreaded driver comes in two flavours - an Intel one and a GNU one. That the latter is missing on some machines is probably a defect of the installation.

Edit: Apparently Intel OpenMP runtime provides a GNU compatibility layer, which means that it could possibly be used as a drop-in replacement of libgomp. At least the symbols are there:

$ nm libiomp5.a | sort | grep GOMP_
0000000000000000 T GOMP_barrier@@VERSION
0000000000000000 T GOMP_barrier@GOMP_1.0
0000000000000000 T __kmp_api_GOMP_barrier
0000000000000000 T __kmp_api_GOMP_barrier_10_alias
...

In that case, what you have to do is:

  • keep -fopenmp while compiling the code so that GCC would recognise the OpenMP pragmas and transform the code into the corresponding calls into libgomp;
  • If GCC is used to link the executable or shared library, do not pass it the -fopenmp option during the linking phase; instead, pass -L/path/to/libiomp5 -liomp5;
  • If GNU ld is used to link the executable/module, replace -lgomp with -liomp5.

If not possible to do the aforementioned changes, the thread on the Intel's forums makes some sense because of the way the linker resolves link-time symbol references, though it is really more of a hack. Passing -Wl,--as-needed forces GNU ld to not emit DT_NEEDED tags for any library that follows it on the command line unless that library satisfies an undefined symbol reference, the assumption being that the GCC driver will insert -lgomp somewhere after the user-supplied options. The idea is to prevent libgomp from being linked with the executable even when there are no unresolved references to GOMP_..., which normally should not be the case since all references, even those from dynamically loaded modules, should be satisfied by libiomp5. Preventing libgomp from being loaded by the RTLD is essential since there are some constructor routines in it that are called no matter if symbols are being imported or not and those do things that might interfere with IOMP.

The linker trick won't work on non-ELF systems like OS X. The Mach-O link editor does not support --as-needed though there might be a different mechanism to achieve the same result on that OS.

Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186
  • Thank you for your reply. I want to make sure I'm understanding you correctly: – Bob Sep 23 '14 at 06:57
  • (little error entering this before) I think you're saying that there's no way to tell gcc not to link libgomp if given the -fopenmp directive, and that gcc will unavoidably send any directives in the code to libgomp rather than libiomp5. Is that right? Regarding MKL, Intel confirmed its not an installation error, there just isn't a libmkl_gnu_thread for some OS's, but also insist that it should be possible to use gcc (although they won't help with how). If I understand you, its just impossible. I can accept that, I just want to make sure I have you right. – Bob Sep 23 '14 at 07:04
  • That Intel statement is really interesting. I just checked the exported symbols of `ligiomp5.a` and indeed there seems to be a GNU compatibility layer exposed. I will update my answer accordingly. – Hristo Iliev Sep 23 '14 at 07:58
  • Thank you... but... what does that mean for my issue? Because I still have binaries that are linking against both libiomp5 and libgomp. I even have binaries that link against both, and then at runtime load multiple plugins some of which link against libiomp5, and some libgomp and some both. I can't get rid of libiomp5 without getting rid of MKL, which I don't want to do. So what would you suggest? (And I do need to link dynamically.) – Bob Sep 23 '14 at 08:18
  • Unless you have control over the options passed to the compiler/linker during the _link_ phase for each software component, there is not much that you could do to prevent this library hell. – Hristo Iliev Sep 23 '14 at 08:34
  • Thanks Hristo -- while I do have control over both the compiling and linking phases, some of the executables in this package want to compile and link themselves with a single command to gcc, which then gets passed -fopenmp from the compiler flags and -liomp5 from the linker flags. I agree Intel's thread about this is very hacky... Maybe I'll give that a try tomorrow. Thank you again. – Bob Sep 23 '14 at 08:59
  • Ok... It turns out that gcc is linked to Apple's ld, which doesn't accept --as-needed. Any other suggestions, or dead end? I can run the Apple install_name_tool to change the links from gomp to iomp5. Then the binary links against libiomp5 *twice*, and to *different versions* (this latter is probably because of library paths environment variables, so may be fixable, but it would still be linked twice). How bad an idea is this? – Bob Sep 23 '14 at 19:41
  • This is becoming fascinating... I just attempted what I described, and it seems to work for most binaries (maybe it will crash later). *But* some of the binaries as for symbol _GOMP_parallel. iomp5 has _GOMP_parallel_start, _GOMP_parallel_end, but not _GOMP_parallel. Thank you so much for your help -- any ideas about next steps from here out? If iomp5 isn't exporting all the gomp sybols... where does that leave anyone trying to compile using gcc against mkl? – Bob Sep 23 '14 at 20:13
  • `--as-needed` is an ELF-specific option. OS X uses the Mach-O binary format whose load commands do not provide the same flexibility as ELF. As for the missing symbols, GCC 4.9 comes with OpenMP 4.0 support and it is quite possible that `libiomp5` does not yet provide compatibility with the newest version of `libgomp`. Could you try it with an older version of GCC on OS X, e.g. 4.8 or 4.7? – Hristo Iliev Sep 24 '14 at 05:49
  • I'm disinclined to. In the first place, I've been linking against the latest libiomp5, which supports OpenMP 4. (Actually, I just checked the website while writing this, and they also claim "drop-in" compatibility with gcc.) In the second -- I don't think the book's going to be worth this big a candle. I really appreciate your help. I'm going to give llvm a try, and if that doesn't solve everything then I'm dumping mkl. If Intel wants the science community to move toward its tools, they can document how to do so. – Bob Sep 24 '14 at 07:35
  • I just checked - the `GOMP_parallel` function is only present in `libgomp` that comes with GCC 4.9. I guess Intel cannot keep their production OpenMP runtime up with the pace at which changes are introduced into GCC. The symbol is present though in the latest version of [the open-source version of the Intel OpenMP runtime](https://www.openmprtl.org/download#stable-releases) which specifically advertises drop-in compatibility with GCC 4.9, i.e. future releases of Intel's compilers ought to be compatible. – Hristo Iliev Sep 24 '14 at 07:51
  • 1) @Bob "I agree Intel's thread about this is very hacky..." which thread? where? I'd like to fix it... – Jim Cownie Sep 24 '14 at 11:23
  • 2) Hristo is basically correct. We try to maintain binary comnpatibility with libgomp in libiomp5. When gcc makes large interface changes we need some time to catch up. (Note that while we've fixed most of the gcc 4.9 changes, we don't yet provide the interfaces needed for task cancellation, because they would have bad performance impact even on codes that don't use them). – Jim Cownie Sep 24 '14 at 11:27
  • 3) A lot of the problems here seem fundamentally related to " Makefiles are not in the greatest shape, and where additional packages from other sources (plug-ins) may be compiled later.". Depending on how hacky you want to be, there is a simple solution here, which is to delete the libgomp.so and replace it with a link to (or copy of) libiomp5.so. If you do that you don't have to change any of your compilation flags. A slightly less hacky approach is to create a link from libiomp5.so to libgomp.so in the intel lib directory and then ensure that LD_LIBRARY_PATH is set appropriately. – Jim Cownie Sep 24 '14 at 11:32
  • p.s. @Hristo. The Intel compiler versions that have the updated Intel OpenMP runtime are listed at https://software.intel.com/en-us/articles/gcc-49-openmp-code-cannot-be-linked-with-intel-openmp-runtime – Jim Cownie Sep 24 '14 at 13:17
  • @Jim: You work for Intel on this project. Having spent a great deal of time going through Intel documentation (all inaccurate or out of date), and trying to get responses from the company --- if you'd like to tell us *how* to solve the problem, please do so. If you'd like to find the info on Intel's website, Hristo and I found it with google. You'll surely do as well. – Bob Sep 24 '14 at 17:06
  • @Jim: as Hristo correctly pointed out, Intel's openmp website says very clearly that iomp5 5.0 has "drop-in compatibility with gcc 4.9": https://www.openmprtl.org/ Don't blame the Makefiles. (You might want to talk to whomever created that link line advisor.) – Bob Sep 24 '14 at 17:16
  • @JimCownie - Now that I'm done venting: Won't a symlink result in two links to the same binary (or two versions of it; the mandatory mkl build scripts put a dir with v 2.0 at the front)? What will that do at runtime, loaded twice by the same bin? Also, is what you're proposing to replace libgomp system-wide? That's hard OSX. Some of Intel's own libraries link to iomp5 by full path, so just putting it in /usr/local/bin won't change the links. Plus, each gcc-made binary full-path-links against a copy of libgomp in that compiler's lib directory. – Bob Sep 24 '14 at 17:22
  • @JimCownie - Finally, I did try something analogous to your symlink, which was to use install-name-tool to change all references to libgomp in this package, once built, to libiomp5. The resulting failures were pretty pervasive and bizarre; I didn't try to do the /usr/local-wide workover that might have cured some. Is this the expected behavior given what I did? Also - it would probably be better if you made your comments a distinct answer. – Bob Sep 24 '14 at 17:25
  • @Bob. 1) You're right, the link advisor is broken. The MKL team will hassle people to get it fixed. 2) Please mail me and we can discuss offline. Nothing secret just that SO thinks this is too long for comments, but won't let you chat, what we're talking about isn't sensibly an answer yet, and the formatting and character limits on comments here are painful. – Jim Cownie Sep 25 '14 at 10:17
7

I think I have an answer at this point; I've had several exchanges with Intel folks, and I want to share the result. This is a mixture of some of their suggestions and what I came up with on my own:

  1. The short answer is, you can't. Gcc wants to force the use of libgomp at the linker stage. If libiomp is also linked, then both libraries will be linked. Which one will be called? I don't know.

  2. The longer answer is that on some distributions, it may be possible to change gcc's default behavior (adding libgomp whenever -fopenmp is set) by creating a custom libgomp.spec or altering the libgomp.spec installed with gcc. On my distribution (homebrew), this was not feasible; the "libgomp.spec" file is empty, and spec's for libgomp are built-in to gcc. All of those would have to be overridden. And this would have to be redone whenever gcc is updated.

  3. It may be possible on some operating systems to replace every copy and link to libgomp, to a symlink to libiomp5. The binary will then have multiple links leading to the same library, under two different names. What will happen then? I don't know.

  4. What I ended-up doing, is moving from gcc to the clang-omp implementation of llvm. It uses libiomp5 unless told otherwise. My concern about this was that part of my project uses fortran, and there's no llvm fortran compiler. It turns out, though, that even when -fopenmp is given to gfortran, so long as llvm is ultimately doing the linking it will wipe-out any references to libgomp and replace them with libiomp5. clang-omp also may have an option to select the omp library with -fopenmp=[libiomp5|libgomp], but I was not able to get this working consistently. Anyway, the clang-omp implementation of llvm 3.5 covers almost all of the openmp spec, and so far it doesn't appear that anything was lost in the switch. In fact performance improved.

  5. I did, for the record, experiment with using gfortran as an llvm frontend using dragonegg. This book was not worth the candle. Dragonegg isn't compatible with gcc 4.9 so it forces gcc 4.8. It was difficult to setup; appears that it would be difficult to maintain as versions change; the llvm folks are not sure how much support dragonegg will have moving forward; and in all events performance was inferior to just using llvm.

  6. The question that drove me here, was how to get a package with C and fortran components, that uses OpenMP, compiled against MKL, which the MKL libraries for my OS are hardlinked against iomp5 and won't accept gomp. The answer to that is that the only viable option was to move from gcc to clang-omp.

  7. This does leave the question, "is iomp5 'drop-in compatible' with gcc 4.9", as claimed on the OpenMP website. The answer is simply, "no," iomp5 and gcc 4.9 will not work with each other --- at least without substantial modifications to one's toolchain, for which no guidance or documentation is available, and it is not clear whether anyone has done this successfully.

Bob
  • 1,274
  • 1
  • 13
  • 26
  • There is a compatibility issue between gcc 4.9 and Intel OpenMP lib. See [link](https://software.intel.com/en-us/articles/gcc-49-openmp-code-cannot-be-linked-with-intel-openmp-runtime) – toddwz Mar 23 '16 at 15:18
4

I'm a technical support engineer on the Intel MKL team. This post has recently come to our attention. The threading layer of Intel MKL does indeed require libiomp5, ie. the Intel OpenMP runtime library. To properly link with libiomp5 using GCC, the code needs to be compiled without using '-fopenmp'. And then the link line needs to explicitly include libmkl_intel_thread and libiomp5.

The current MKL documentation does not provide enough explanations on this usage model. And the MKL link line advisor is outright broken. I'm sorry for all the inconvenience and confusions. We will get the link line advisor fixed as soon as possible, and will improve the User's Guide to better help GCC users on OS X.

Zhang Zhang
  • 41
  • 1
  • Thank you! But... isn't libiomp5 supposed to be "drop-in" compatible with gcc? If we don't use the -fopenmp directive, aren't we turning-off openmp throughout all of the *non*-MKL code? – Bob Sep 26 '14 at 00:29
  • Right now, about half the home-compiled part of my system links both libiomp5 and libgomp. Jim Crownie's suggestion to symlink iomp5 to libgomp, if that would work, might remedy some of the situation. Would it? – Bob Sep 26 '14 at 02:14
  • 1
    Everything is broken these days. We should build all the software from scratch. But thank you for the advice. – mip Apr 28 '20 at 18:29