Makefile linking using g++ compiler with CUDA object files

Question

I am trying to compile cuda object files with nvcc, and compile the final main script using the g++ compiler. I have seen this post but wasn't able to get my example working. The error I am getting seems to be a linkage error:

nvcc -c -g -I -dlink -I/usr/local/cuda-11/include -I. -L/usr/local/cuda-11/lib64 -lcudart -lcurand module.cu -o module.o 
g++ -I/usr/local/cuda-11/include -I. -L/usr/local/cuda-11/lib64 -lcudart -lcurand module.o main.cpp -o main
module.o: In function `call_kernel()':
/home/ubuntu/Desktop/CUDA/MPI&CUDA/module.cu:16: undefined reference to `__cudaPushCallConfiguration'
module.o: In function `__cudaUnregisterBinaryUtil()':
/usr/local/cuda-11/include/crt/host_runtime.h:259: undefined reference to `__cudaUnregisterFatBinary'
module.o: In function `__nv_init_managed_rt_with_module(void**)':
/usr/local/cuda-11/include/crt/host_runtime.h:264: undefined reference to `__cudaInitModule'

What am I doing wrong ? I am aware I could simply compile main.cpp with nvcc but it is something I don't want, as in my problem, I will replace g++ with mpicxx later and have MPI code inside my main.cpp script.

My makefile is:

INC := -I$(CUDA_HOME)/include -I.
LIB := -L$(CUDA_HOME)/lib64 -lcudart -lcurand
CUDAFLAGS=-c -g -I -dlink $(INC) $(LIB)

all: main

main: module.o
    g++ $(INC) $(LIB) module.o main.cpp -o main

module.o: module.cu module.h 
    nvcc -c -g -I -dlink $(INC) $(LIB) module.cu -o module.o 

clean: 
    rm -rf *.o

main.cpp

#include "module.h"

int main(){

    return 0;
}

module.cu

#ifdef __CUDACC__
#define CUDA_GLOBAL __global__
#else
#define CUDA_GLOBAL
#endif

#include <cuda.h>
#include "module.h"

CUDA_GLOBAL
void kernel(){

}

void call_kernel(){
    kernel<<<1,1>>>();
}

module.h

#ifndef _MODULE_H_
#define _MODULE_H_

#ifdef __CUDACC__
#define CUDA_GLOBAL __global__
#else
#define CUDA_GLOBAL
#endif

#include <numeric>
#include <cuda.h>

CUDA_GLOBAL
void kernel();

void call_kernel();

#endif

Answer 1

Your link line is wrong. All libraries (e.g., -lfoo) must come at the end of the link line after all the object files (e.g., .o files).

Not only that, but they need to be ordered properly (but I have no idea what the right order is so maybe they are correct above).

Almost all modern linkers are "single pass" linkers which means that they only go through the libraries one time, and since they only pull symbols in that they already need you must order your libraries with the "highest level" content first, and the "lower level" content following.

Answer 2

Thanks to the explanations of MadScientist, the linkages must be done at the end of the link line, after all .o files:

INC := -I$(CUDA_HOME)/include -I.
LIB := -L$(CUDA_HOME)/lib64 -lcudart -lcurand

all: main

main: module.o
    mpicxx module.o main.cpp -o main $(INC) $(LIB)

module.o: module.cu module.h 
    nvcc -c -g module.cu -o module.o -I -dlink

clean: 
    rm -rf *.o