0

I recently tried to build my https://github.com/eyalroz/cuda-api-wrappers/ library's examples after switching to another Linux distribution on the same machine. Strangely enough, I encountered a linking issue. The command:

/usr/bin/c++   -Wall -std=c++11 -g  CMakeFiles/device_management.dir/examples/by_runtime_api_module/device_management.cpp.o -o examples/bin/device_management -rdynamic lib/libcuda-api-wrappers.a -Wl,-Bstatic -lcudart_static -Wl,-Bdynamic -lpthread -ldl -lrt

fails to find the CUDA runtime library, and I get:

CMakeFiles/device_management.dir/examples/by_runtime_api_module/device_management.cpp.o: In function `cuda::device::peer_to_peer::get_attribute(cudaDeviceP2PAttr, int, int)':
/home/eyalroz/src/mine/cuda-api-wrappers/src/cuda/api/device.hpp:38: undefined reference to `cudaDeviceGetP2PAttribute'
collect2: error: ld returned 1 exit status

but if I add -L/usr/local/cuda/lib64 it builds fine. This didn't use to happen before; and it doesn't happen on another machine I've checked on, nor does it even happen to other targets using the CUDA runtime in the same CMakeLists.txt (like version_managament).

FindCUDA seems to be finding everything, as the value of ${CUDA_LIBRARIES} is /usr/local/cuda/lib64/libcudart_static.a;-lpthread;dl;/usr/lib/x86_64-linux-gnu/librt.so. And the target lines in CMakeLists.txt are:

add_executable(device_management EXCLUDE_FROM_ALL examples/by_runtime_api_module/device_management.cpp)
target_link_libraries(device_management cuda-api-wrappers ${CUDA_LIBRARIES})

as is suggested in answers to other related questions (e.g. here). Why is this happening? Should I "manually" add the -L switch?

Edit: Following @RobertCrovella's suggestion, here are the ld search paths:

$ gcc -print-search-dirs | sed '/^lib/b 1;d;:1;s,/[^/.][^/]*/\.\./,/,;t 1;s,:[^=]*=,:;,;s,;,;  ,g' | tr \; \\012 | tr ':' "\n" | tail -n +3
  /usr/local/cuda/lib64/x86_64-linux-gnu/5/
/usr/local/cuda/lib64/x86_64-linux-gnu/
/usr/local/cuda/lib/
/usr/lib/gcc/x86_64-linux-gnu/5/
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/5/
/usr/x86_64-linux-gnu/lib/x86_64-linux-gnu/
/usr/x86_64-linux-gnu/lib/
/usr/lib/x86_64-linux-gnu/5/
/usr/lib/x86_64-linux-gnu/
/usr/lib/
/lib/x86_64-linux-gnu/5/
/lib/x86_64-linux-gnu/
/lib/
/usr/lib/x86_64-linux-gnu/5/
/usr/lib/x86_64-linux-gnu/
/usr/lib/
/usr/local/cuda/lib64/
/usr/x86_64-linux-gnu/lib/
/usr/lib/
/lib/
/usr/lib/

$ ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012
SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu")
SEARCH_DIR("=/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")
SEARCH_DIR("=/usr/x86_64-linux-gnu/lib64")
SEARCH_DIR("=/usr/x86_64-linux-gnu/lib")

Notes:

  • Yes, I know the CMakeLists.txt there is ugly.
Community
  • 1
  • 1
einpoklum
  • 118,144
  • 57
  • 340
  • 684
  • 2
    possibly (unlikely?) your `ld` linker search paths on the two machines are configured differently. IMO, having the `ld` linker seach path (not the dynamic library load path) set up to automatically search `/usr/local/cuda/lib64` would be *unusual*, so I think the best suggestion is to explicitly identify `-L /desired/path/to/cuda/libs`. You could use one of the methods suggested [here](http://stackoverflow.com/questions/9922949) (e.g. in 3rd answer) to compare the `ld` linker search paths (**not** the dynamic library load path) between the two machines, to prove or disprove this theory. – Robert Crovella Jan 01 '17 at 15:29
  • Assuming the first `c++` command line you have shown is issued **identically** between the passing and failing machines, then this is not/should not be a CMake issue, I don't think. At least that difference in behavior is not explainable by any CMake configuration. Therefore, investigation of the difference in behavior may follow the path indicated in my previous comment. – Robert Crovella Jan 01 '17 at 15:33
  • Leaving aside the fact that the library is question happens to be the CUDA runtime API library, what exactly makes this a question about CUDA? Isn't it just "cmake and g++ don't find a library on one system, but do on another notionally identical system. Why?" – talonmies Jan 01 '17 at 15:38
  • @RobertCrovella: Unfortunately I can't issue commands on the OS I replaced :-( Also, see edit - the ld search path doesn't include the CUDA library directory, but gcc does include it by looking at the environment I suppse. – einpoklum Jan 01 '17 at 15:41
  • @talonmies: It's not a question about programming in CUDA, it's about building CUDA code; it involves the CUDA packages and/or installer and the CMake CUDA module. I think that qualifies. – einpoklum Jan 01 '17 at 15:43
  • I suppose its possible, then, that the linker search path on the machine that was wiped out may have included the necessary path so that the command issued would find the lib in question. Without being able to actually inspect or test that machine, it would seem that this is all based on conjecture. That theory would (I think) explain the observed behavior, assuming `/usr/bin/c++` is symlinked to the gnu toolchain. Again, the **observed behavior** of the first command line you have shown (assuming it is/was issued **identically** on both machines) should not be dependent on `CMake` AFAIK. – Robert Crovella Jan 01 '17 at 15:52
  • @RobertCrovella: The point is, I was assuming what I had in the `CMakeLists.txt` - that is, a FindCUDA which succeeds and the lines I used for defining a target - should be enough, given a CUDA installation, without the user invokving `cmake . && make examples` having to make any additional settings. That is, I was assuming the observed behavior *should* be dependent on CMake in the sense of CMake being able coerce the appropriate observed behavior. – einpoklum Jan 01 '17 at 16:06
  • 1
    If you want `CMake` to issue commands which are well suited for a variety of CUDA installs, then my suggestion would be to force `CMake` to issue the `c++` command that is doing the linking in a sensible way - and that is to explicitly include the `-L` switch I mentioned already. If that is your intent, then the fact that this happened to work as-is on another machine is a red herring, IMO, and you should craft your question to ask "how can I get CMake to add the necessary `-L` switch to this command line in this scenario?" – Robert Crovella Jan 01 '17 at 16:10
  • I'm not a `CMake` expert, so I can't tell you what is the "best" way to do this, but I do think there are very simple options in `CMakeLists.txt` which will allow you to add arbitrary `-L` switches. – Robert Crovella Jan 01 '17 at 16:14
  • @RobertCrovella: So I think the answer I added summarizes things. – einpoklum Jan 01 '17 at 16:24
  • @einpoklum: My point was that the first command you have shown should probably fail on every system, because there is no explicit library search path for where the CUDA library is. The reason it works on the system you showed in the edit is because someone/thing has modified the spec file or set `LIBRARY_PATH` for gcc to include the path where CUDA libraries are found. I was trying to suggest that maybe you are looking at this from the wrong angle and in the wrong place -- try understanding why it works rather than why it doesn't work, which doesn't have anything to do with CUDA per se. – talonmies Jan 01 '17 at 16:36
  • @talonmies: Ok. And - I did have the CUDA library path in `LIBRARY_PATH` - on both systems actually; it doesn't help somehow. I'll try to expand the answer below to reflect what you said. – einpoklum Jan 01 '17 at 17:08

1 Answers1

0

TL;DR:

After the FindCUDA invocation, add the lines:

get_filename_component(CUDA_LIBRARY_DIR ${CUDA_CUDART_LIBRARY} DIRECTORY)
set(CMAKE_EXE_LINKER_FLAGS ${CMAKE_EXE_LINKER_FLAGS} "-L${CUDA_LIBRARY_DIR}")

and building should succeed on both systems.

Discussion:

(Paraphrasing @RobertCrovella and myself in the comments:)

OP was expecting, that if the following hold:

  • FindCUDA succeeds
  • ${CUDA_LIBRARIES} includes a valid full path to either the static or the dynamic CUDA runtime library
  • the library dependency is indicated using target_link_libraries(relevant_target ${CUDA_LIBRARIES})

... then the CMake-based build he was attempting should succeed on a variety of valid CUDA installations. That is (unfortunately) not the case, since while FindCUDA does locate the CUDA library path, it does not actually make your linker search that path. So a failure should actually be expected. The build had worked on OP's old system due to a "fluke", or rather, due to OP having added the CUDA library directory to the linker's search path, somehow, apriori.

The linking command must be issued with the -L/path/to/cuda/libraries switch, so that the linker knows where to looks for the (unspecified-path) libraries referred to be the CUDA-related -l switches (in OP's case, -lcudart_static).

This answer discusses how to do that in CMake for different kinds of targets. You might also want to have a look at man gcc (the GCC manual page, also available here) regarding the -l and -L options, if you are not familiar with them.

Community
  • 1
  • 1
einpoklum
  • 118,144
  • 57
  • 340
  • 684
  • @talonmies Done, sorry. Was it you who's been downvoting a bunch of my questions today? – einpoklum Feb 11 '17 at 11:04
  • Absolutely not. I only noticed this one when it popped up in the active questions list because of your edit. if you have been serial downvoted by someone, the system will probably detect it and reverse it with the next 24 hours. – talonmies Feb 11 '17 at 11:11
  • @talonmies: Ok, sorry. It's just that there were 3 downvotes on around-0-vote questions, at least two of which with my own answer, and then I saw your comment here. – einpoklum Feb 11 '17 at 12:10