0

Consider the following CUDA program, in a file named foo.cu:

#include <cooperative_groups.h>
#include <stdio.h>

__global__ void my_kernel() {
    auto g = cooperative_groups::this_grid();
    g.sync();
}

int main(int, char **) {
    cudaLaunchCooperativeKernel( (const void*) my_kernel, 2, 2, nullptr, 0, nullptr);
    cudaDeviceSynchronize();
}

This program needs to be compiled with -rdc=true (see this question); and needs to be explicitly linked against libcudadevrt. Ok, no problem... or is it?

$ nvcc -rdc=true -o foo  -gencode arch=compute_61,code=sm_61 foo.cu  -lcudadevrt
nvlink error   : Undefined reference to 'cudaCGGetIntrinsicHandle' in '/tmp/tmpxft_000036ec_00000000-10_foo.o'
nvlink error   : Undefined reference to 'cudaCGSynchronizeGrid' in '/tmp/tmpxft_000036ec_00000000-10_foo.o'

Only if I explicitly add the library's folder with -L/usr/lib/x86_64-linux-gnu, is it willing to build my program.

This is strange, because all of the CUDA libraries on my system are in that folder. Why isn't NVCC/nvlink looking in there?

Notes:

  • I'm using Devuan GNU/Linux 3.0.
  • CUDA 10.1 is installed as a distribution package.
  • An x86_64 machine with a GeForce 1050 Ti card.
talonmies
  • 70,661
  • 34
  • 192
  • 269
einpoklum
  • 118,144
  • 57
  • 340
  • 684
  • Devuan isn't a supported distro for CUDA development. On Ubuntu (for example), using [packages for installation directly from NVIDIA](https://developer.nvidia.com/cuda-downloads), I have no need to add the `-L` switch indicated here, to get this to work. – Robert Crovella Dec 25 '19 at 21:03
  • @RobertCrovella: Still - what paths is nvlink searching? Is there an environment variable I could set perhaps? – einpoklum Dec 25 '19 at 21:11
  • 1
    Even if I could answer a question like this, I think it would be unwise to do so. It could communicate to other future readers that this might be a safe or sensible thing to do. I don't consider it to be. My suggestion would be to switch to a supported development environment. – Robert Crovella Dec 25 '19 at 21:22
  • @RobertCrovella: I now [see this link failure](https://travis-ci.com/github/eyalroz/cuda-api-wrappers/jobs/522189055) on Ubuntu Bionic, which _is_ a supported distribution. – einpoklum Jul 06 '21 at 21:21
  • I just tried CUDA 11.2 on Ubuntu 20.04. Your code in this question compiles cleanly, using the command you indicate. Then I tried [this container](https://hub.docker.com/layers/nvidia/cuda/10.1-devel-ubuntu18.04/images/sha256-03fddd924b124be110d0b48b74b9228f223785d042b6894df6668e028c5cf159?context=explore). It also compiles cleanly. So I'm not able to reproduce that observation. I'm not able to debug based on that travis output. It's clearly an environment issue of some sort. If you can develop a concise set of instructions that go from a clean OS install to show the issue, perhaps file a bug – Robert Crovella Jul 07 '21 at 01:47

1 Answers1

1

NVCC, or perhaps nvlink, looks for paths in an environment variable named LIBRARIES. But - before doing so, the shell script /etc/nvcc.profile is executed (at least, it is on Devuan).

On Devuan 3.0, that file has a line saying:

LIBRARIES   =+ $(_SPACE_) -L/usr/lib/x86_64-linux-gnu/stubs

so that's where your NVCC looks to by default.

You can therefore do one of two things:

  1. Set the environment variable outside NVCC, e.g. in your ~/.profile or ~/.bashrc file:

    export LIBRARIES=-L/usr/lib/x86_64-linux-gnu/

  2. Change that nvcc.profile line to say:

    LIBRARIES   =+ $(_SPACE_) -L/usr/lib/x86_64-linux-gnu -L/usr/lib/x86_64-linux-gnu/stubs

and NVCC will successfully build your binary.

einpoklum
  • 118,144
  • 57
  • 340
  • 684