Runtime linker error using Thrust in MATLAB MEX file

Question

I'm having trouble using the CUDA Thrust library in MATLAB MEX code.

I have an example that runs fine externally, but if I compile and run it as a MEX file, it produces "missing symbol" errors at runtime.

It seems specific to the Thrust library. If instead of thrust::device_vector I use cudaMalloc with cudaMemcpy or cublasSetVector then everything is fine.

Minimum example

thrustDemo.cu:

#ifdef MATLAB_MEX_FILE
    #include "mex.h"
    #include "gpu/mxGPUArray.h"
#endif
#include <thrust/device_vector.h>
#include <vector>

void thrustDemo() {
    std::vector<double> foo(65536, 3.14);
    thrust::device_vector<double> device_foo(foo);
}

#ifdef MATLAB_MEX_FILE
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, mxArray const *prhs[]) {
    thrustDemo();
}
#else
int main(void) { thrustDemo(); }
#endif

The problem

I can compile this from the command line (nvcc thrustDemo.cu) and run the resulting executable just fine.

When I try to build this as a MATLAB MEX file (mexcuda thrustDemo.cu from within MATLAB R2017a), it compiles and links just fine:

>> mexcuda thrustDemo.cu
Building with 'nvcc'.
MEX completed successfully.

But when I try to run it, I get the following error:

>> thrustDemo()
Invalid MEX-file '/home/kqs/thrustDemo.mexa64':
Missing symbol '_ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE5c_strEv' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE5emptyEv' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt12length_errorC1EPKc' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt13runtime_errorC2EPKc' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEaSEPKc' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1EPKcRKS3_' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1ERKS4_' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1Ev' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEpLEPKc' required by '/home/kqs/thrustDemo.mexa64'
Missing symbol '_ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEpLERKS4_' required by '/home/kqs/thrustDemo.mexa64'.

This is pretty foreign to me; can somebody tell me what this means? These look like linker errors, but they're being generated at runtime. Also, I thought Thrust was a template library, so what is there to link to?

And finally, replacing thrust::device_vector with cudaMalloc and either cudaMemcpy or cublasSetVector works just fine. So for now I'm stuck with a bunch of cudaMalloc in my code, which seems...distasteful. I'd really like to be able to use Thrust.

Versions

MATLAB R2017a

nvcc V8.0.61, gcc 5.4.0, Ubuntu 16.04.2

NVidia driver 375.39, GTX 1060 graphics card (Compute Capability 6.1)

Update: `ldd` output

Per comments, I checked the dependencies of the MEX file using ldd thrustDemo.mexa64:

linux-vdso.so.1 =>  (0x00007ffdd35ea000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f097eccf000)
libcudart.so.8.0 => /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0 (0x00007f097ea69000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f097e852000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f097e489000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f097e180000)
/lib64/ld-linux-x86-64.so.2 (0x0000562df178c000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f097df7b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f097dd5e000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f097db56000)

I tried looking for one of these missing symbols, and was able to find it:

$ nm -D /usr/lib/x86_64-linux-gnu/libstdc++.so.6 | grep "_ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE5c_strEv"
0000000000120be0 W _ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE5c_strEv

So it seems that MATLAB must be looking in the wrong place.

It is not a runtime error. The error is occurring when the mex file is being loaded. I do not know the cause of the error. But you should be able to inspect your mex file using tools like ldd in linux to check dependencies. — Navan, Apr 26 '17 at 20:41
This is some sort of broken C++/stdlib problem or host compiler mismatch. The functions involved are `std::__cxx11::basic_string, std::allocator >::c_str() const` which have nothing to do with CUDA — talonmies, Apr 26 '17 at 20:48
I see; the error output makes a lot more sense now. I think MATLAB prefers to use its own version of `libstdc++`, which may be the root cause. Thank you both for your comments. — KQS, Apr 26 '17 at 21:04

KQS · Accepted Answer · 2017-10-27T19:05:57.757

It turns out this has nothing to do with Thrust but is rather an issue with MATLAB having its own version of the C++ standard libraries.

Thanks to @Navan and @talonmies for their helpful comments.

Interpreting the error

First, MATLAB is raising these errors when it's loading the MEX file. The MEX file has external dependencies, and MATLAB could not find them.

After checking these dependencies with the Linux utility ldd, then using nm to list the symbols define by these libraries, I found that the system version of the libstdc++ shared library actually contains these "missing symbols". Hence why the externally-compiled version works just fine.

Solving the issue

The root problem, then, is that MATLAB ships with its own, older version of libstdc++ that is lacking these functions. Knowing the root cause, I found questions like these:

How to tell mex to link with the libstdc++.so.6 in /usr/lib instead of the one in the MATLAB directory?

Version GLIBCXX_3.4.11 not found (required by buildW.mexglx)

which describe workarounds that were indeed successful for my problem.

In particular, I used LD_PRELOAD when launching MATLAB to force MATLAB to use the system libstdc++ instead of its own copy:

$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/local/MATLAB/R2017a/bin/matlab

Update: a better solution

It turns out that the folks at GCC are well aware of this incompatibility and discuss it here:

In the GCC 5.1 release libstdc++ introduced a new library ABI that includes new implementations of std::string and std::list. These changes were necessary to conform to the 2011 C++ standard which forbids Copy-On-Write strings and requires lists to keep track of their size.

In order to maintain backwards compatibility for existing code linked to libstdc++ the library's soname has not changed and the old implementations are still supported in parallel with the new ones. ... The _GLIBCXX_USE_CXX11_ABI macro (see Macros) controls whether the declarations in the library headers use the old or new ABI.

To tell gcc to use the older ABI, we just need to define _GLIBCXX_USE_CXX11_ABI as 0 before we include any library headers, e.g. by passing a -D option to the compiler:

-D_GLIBCXX_USE_CXX11_ABI=0

For the sake of completeness, I'll mention that my full mexcuda call looks like this:

nvcc_opts = [...
    '-gencode=arch=compute_30,code=sm_30 ' ...
    '-gencode=arch=compute_50,code=sm_50 ' ...
    '-gencode=arch=compute_60,code=sm_60 ' ...
    '-std=c++11 ' ...
    '-D_GLIBCXX_USE_CXX11_ABI=0 '   % MATLAB's libstdc++ uses old library ABI
    ];
mexcuda_opts = {
    '-lcublas'                      % Link to cuBLAS
    '-lmwlapack'                    % Link to LAPACK
    '-lcufft'                       % Link to cuFFT
    ['NVCCFLAGS="' nvcc_opts '"']
    '-L/usr/local/cuda/lib64'       % Location of CUDA libraries
    };
mexcuda(mexcuda_opts{:}, src_file);