Code jumps into unreferenced shared object

Question

My statically linked CryptoPP code (when called via Matlab mex on Linux) jumps into the libmwflcryptocryptopp.so binary (and then it freezes).

How can my code jump into a foreign .so instead of the statically linked library?

from function CryptoPP::SourceTemplate<CryptoPP::FileStore>::PumpAll2 in my vcpkg_installed/.../cryptopp/filters.h:1443 it jumps into CryptoPP::BufferedTransformation::TransferAllTo2 from Matlabs own libmwflcryptocryptopp.so file.

the gdb stack trace

#0  0x00007ffff02e6cab in CryptoPP::BufferedTransformation::Peek(unsigned char&) const () from /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
#1  0x00007ffff02e6c19 in CryptoPP::BufferedTransformation::AnyRetrievable() const () from /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
#2  0x00007ffff02e6ffe in CryptoPP::BufferedTransformation::TransferMessagesTo2(CryptoPP::BufferedTransformation&, unsigned int&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
#3  0x00007ffff02e7129 in CryptoPP::BufferedTransformation::TransferAllTo2(CryptoPP::BufferedTransformation&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) () from /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so
#4  0x00007fff9758aa4d in CryptoPP::SourceTemplate<CryptoPP::FileStore>::PumpAll2 (this=0x7fffdd87fbc0, blocking=<optimized out>)
    at /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/filters.h:1443
#5  0x00007fff9758a130 in CryptoPP::Source::PumpAll (this=0x7fffdd87fbc0) at /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/filters.h:1420
#6  CryptoPP::Source::SourceInitialize (parameters=warning: RTTI symbol not found for class 'CryptoPP::AlgorithmParameters'
..., pumpAll=true, this=0x7fffdd87fbc0) at /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/filters.h:1420
#7  CryptoPP::FileSource::FileSource (attachment=0x7fffd6063160, pumpAll=true, in=..., this=0x7fffdd87fbc0) at /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/files.h:102
#8  mexFunction (nlhs=<optimized out>, plhs=<optimized out>, nrhs=<optimized out>, prhs=<optimized out>) at /home/keinkoenig/src/CryptoppMinimal/CryptoppMinimal.cpp:18

minmal example code

#include <mex.h>
#include <cryptopp/files.h>
#include <cryptopp/aes.h>
#include <cryptopp/modes.h>
#include <fstream>

unsigned char Key[CryptoPP::AES::DEFAULT_KEYLENGTH] = { 0x00,0x11,0x22,0x33,0x44,0x55,0x66,0x77,0x88,0x99,0xAA,0xBB,0xCC,0xDD,0xEE,0xFF };
unsigned char IV[CryptoPP::AES::BLOCKSIZE] = { 0x00,0x11,0x22,0x33,0x44,0x55,0x66,0x77,0x88,0x99,0xAA,0xBB,0xCC,0xDD,0xEE,0xFF };

extern "C" 
{
    void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
    {
        std::ifstream instream(mxArrayToString(prhs[0]));

        std::string decryptedString;
        CryptoPP::CFB_Mode<CryptoPP::AES>::Decryption cfbDecryption(Key, sizeof(Key), IV);
        CryptoPP::FileSource(instream, true, new CryptoPP::StreamTransformationFilter( cfbDecryption, new CryptoPP::StringSink( decryptedString)));
    }
}

and the CMakeLists linking

set(PROJECT_NAME CryptoppMinimal)
...
target_link_libraries(${PROJECT_NAME} PRIVATE cryptopp-static)

starting console matlab with gdb

/usr/local/MATLAB/R2020b/bin/matlab -nojvm -Dgdb

and running the test mex function in Matlab

addpath ~/build-o1/bin
CryptoppMinimal('/home/keinkoenig/src/encrypted.dat')

Of note is that the .so appears to be the one shipped with matlab and not the one from "your" version of cryptocpp. If the library was compiled with `-fPIC`, and Matlab has its cryptocpp.so loaded in memory before loading your code, it would cause the preloaded symbols to take priority over the symbols from your project. — , Nov 04 '21 at 15:39

score 1 · Accepted Answer · 2021-11-04T15:58:24.663

TL;DR; your cryptocpp is getting LD_PRELOADed by Matlab, even if it is being linked statically.

A big hint towards what is going on is the path of the .so being loaded:

#3  0x00007ffff02e7129 [...] /usr/local/MATLAB/R2020b/bin/glnxa64/libmwflcryptocryptopp.so

Compared to the headers that were used during compilation:

#5 [...] /home/keinkoenig/build-o1/vcpkg_installed/x64-linux/include/cryptopp/filters.h:1420

What this shows is that not only is cryptocpp running from a dynamic library, it's running from a different .so than the one you compiled against!

How could this happen? If a static library is compiled with position independent code, calls to functions in the library will still be dispatched in the same way they would for a dynamic library: with a jump table.

And much like dynamic libraries, if the jump table is already populated when the code loaded, the existing function pointers will be reused.

So, if:

crypto-cpp was compiled with -fPIC
Matlab happens to have a crypto-cpp .so loaded prior to your code being loaded.

Then the version of crypto-cpp embedded within your project will be ignored, and Matlab's will be used instead.

Having a quick look at the crypto-cpp project, we find, in the Makefile:

# Add -fPIC for targets *except* X86, X32, Cygwin or MinGW
ifeq ($(IS_X86)$(IS_CYGWIN)$(IS_MINGW),000)
  ifeq ($(findstring -fpic,$(CXXFLAGS))$(findstring -fPIC,$(CXXFLAGS)),)
    CRYPTOPP_CXXFLAGS += -fPIC
  endif
endif

Which appears to confirm this is what's going on.

You can confirm that by loading and running your library from outside Matlab, and it should be calling the static library's code as expected.

preloading my binary crashes matlab on start. so there seems to be an incompatibility of the cryptopp versions. is there any way to fix this? — keinkoenig, Nov 04 '21 at 16:25
From a quick search, it seems like this is some Matlab-specific fork of the library. You could start here: https://stackoverflow.com/questions/678254/ for a fix. The `dlopen()`/ `dlsym()` approach would be the simplest if it works for you. — , Nov 04 '21 at 18:36

Code jumps into unreferenced shared object

1 Answers1