undefined reference to `cuMemAlloc_v2'

Question

I need to modify linux rdma-core driver and add some cuda-related functions; firstly, cuMemAlloc. I have changed the CMake file in the driver to include the cuda.h header file. But when I compile the driver, I get undefined reference to error

In the CMakeLists.txt file, I have included

project(rdma-core LANGUAGES CUDA C CXX)
FIND_PACKAGE(CUDA 12.2 REQUIRED)

and -I/usr/local/cuda-12.2/include -lcuda to

RDMA_AddOptCFlag(CMAKE_C_FLAGS HAVE_C_WARNINGS
  "-Wall -Wextra -Wno-sign-compare -Wno-unused-parameter -I/usr/local/cuda-12.2/include -lcuda")

Although cuda.h is successfully found, I get the following compilation error:

/usr/bin/ld: CMakeFiles/mlx5.dir/verbs.c.o: in function `mlx5_alloc_dyn_uar':
/home/nurlan/rdma-core/providers/mlx5/verbs.c:235: undefined reference to `cuMemAlloc_v2'
/usr/bin/ld: /home/nurlan/rdma-core/providers/mlx5/verbs.c:235: undefined reference to `cuMemAlloc_v2'
/usr/bin/ld: CMakeFiles/mlx5.dir/verbs.c.o: in function `mlx5_alloc_pd':
/home/nurlan/rdma-core/providers/mlx5/verbs.c:153: undefined reference to `cuMemAlloc_v2'
collect2: error: ld returned 1 exit status
make[2]: *** [providers/mlx5/CMakeFiles/mlx5.dir/build.make:501: lib/libmlx5.so.1.24.48.0] Error 1
make[1]: *** [CMakeFiles/Makefile2:1560: providers/mlx5/CMakeFiles/mlx5.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Anyone encountering similar problem before or knowing the solution to this problem?

Answer 1

_{Expanding on @talonmies valid comment/answer}

Part I: The lower-level error and solution

When you get an error of the form:

/usr/bin/ld: somefile.o: in function `func1`... undefined reference to `func2`

this is a linker error. That means the compilation of your source files has succeeded (and, in particular, relevant header files were found); but the linking of the resulting object files into an executable has failed, since no compiled code has been found for a function named func2.

That can mean one of several things, but perhaps the most likely cause is that the linker just hasn't been directed to where the relevant library is located. It has the files you have compiled, but even when you give it the relevant library names with -lfoo for each library - it still needs to know where to look for the file libfoo.a (or libfoo.so etc). The linker can be told that either using -L/path/to/library/files/ command-line arguments, or via the LIBRARY_PATH environment variable.

In your case, the cuMemAlloc() is a CUDA Driver API function. So, you need to mention the directory in which your CUDA driver library access library is located. Typically, the driver library is at /usr/lib/x86_64-linux-gnu/libcuda.so (depends on your platform of course) - if it is installed. But the CUDA distribution itself has a stub driver library, at /usr/local/cuda-12.2/lib64/stubs/libcuda.so (assuming that's where you installed CUDA), which you should also be able to link against. So, your linker command line needs a -L/usr/lib/x86_64-linux-gnu or something like a -L/usr/local/cuda-12.2/lib64/stubs argument.

Part 2: Fixing things within CMake

When using CMake to configure your build, you achieve the effects described above with a higher level of abstraction and less detail. That is, you mostly avoid setting specific command-line arguments yourself, and rather define appropriate target dependencies.

First - the CUDA package is deprecated in newer versions of CMake! See the explanation in this answer. So, instead of:

find_package(CUDA 12.2 REQUIRED)

we write:

find_package(CUDAToolkit 12.2 REQUIRED)

(see the documentation for this package.) Then we can use the targets that provides us:

target_link_libraries(some_rdma_target CUDA::cuda_driver)

or

target_link_libraries(some_other_rdma_target CUDA::cudart)

(or both, depending on which target) This should take care of all of the -I command-line arguments for the compilation and the -l and -L command-line arguments for the linking. You should not have to add any of these yourself to the CMAKE_C_FLAGS or or CMAKE_CXX_FLAGS.