4

I have a MEX file compiled normally with g++.

I recently changed its compilation to use clang++, and included -fsanitize=address into both the LDFLAGS and CFLAGS (note: no CXX flags exist for this project).

However, despite this, once I get to MATLAB and attempt to call the MEX file, I get:

Invalid MEX-file '(path to mex file)': undefined symbol: __asan_option_detect_stack_use_after_return.

That specific error is really common when people mess up linking in the address sanitizer correctly. However, in literally every file I'm compiling it's linked in. Every .o, every .mexa64.

I suspect this is because MATLAB itself isn't capable of that, but I'm unsure. Some guidance from other MEX developers would be fantastic.

Full steps I used for anyone who needs this:

1: Install libasan (for me it was "yum install libasan", but it might vary)

2: Add -fsanitize=address to the LDFLAGs and CFLAGs of the makefile building the MEX files and object files for my project.

3: Make clean and make to ensure it has the library included (I built with g++, apparently clang defaults to the STATIC version of libasan, which won't work in situations like this where the running executable isn't actually compiled with libasan)

4: In a terminal, do:

export LD_PRELOAD=/lib64/libasan.so.5 (this location varies though. I found out how to locate it using this post: Get location of libasan from gcc/clang)

Then do:

export ASAN_OPTIONS=halt_on_error=false

5: Finally, call MATLAB:

matlab -nojvm -nodesktop -nosplash

6: Then (and this might be specific to my project) I CD'd into the directory where the MATLAB project was, did addpath(genpath('.')) to add all its files, and finally called the actual MATLAB script that does the work.

The result was errors in green and red, like:

Address 0x(some address) is located in stack of thread T(thread number) SUMMARY: AddressSanitizer: memcpy-param-overlap (libasan.so.5+(some number)) or ERROR: AddressSanitizer: memcpy-param-overlap: memory ranges [range] and [range] overlap

Tyler Shellberg
  • 1,086
  • 11
  • 28
  • Are you using a shared object asan library, or linking it statically? https://github.com/google/sanitizers/wiki/AddressSanitizerAsDso – Cris Luengo Jul 10 '19 at 18:40
  • Also, if you use `g++` or `clang++` to compile, you should be using `CXXFLAGS`, not `CFLAGS`. – Cris Luengo Jul 10 '19 at 18:42
  • @CrisLuengo I believe it's the static. I did "sudo dnf install libasan libasan-static". Also had to do "sudo yum install libasan" – Tyler Shellberg Jul 10 '19 at 18:43
  • 1
    Try `ldd mexfile.mexa64`. It will report which shared object libraries are linked to. It will tell you it cannot find the MATLAB libraries (ignore that), pay attention to libasan: is it looking for it? If yes, then you're using the dynamic version. If not, you're using a static version. – Cris Luengo Jul 10 '19 at 18:45
  • @CrisLuengo Then it looks like I'm using the static. I don't see anything about libasan. Should I be using the dynamic instead? If so, how wouldI switch? – Tyler Shellberg Jul 10 '19 at 18:49
  • @CrisLuengo I'm starting to follow I think. So the static version only works if the executable itself is linked with it. Since MATLAB is not, it won't work. To do so dynamically...I'd have to uninstall ASAN, get the source, re-build it with the appropriate flag (```DCOMPILER_RT_BUILD_SHARED_ASAN=ON```), and then re-build my project linking against that...somehow. Does that seem about right? – Tyler Shellberg Jul 10 '19 at 19:16
  • 1
    GCC uses the dynamic libasan by default, you could try using `g++` to compile. – Cris Luengo Jul 10 '19 at 19:20
  • @CrisLuengo Huh. Yeah, that seems to work. Though now I get ```ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD```. I'll give a shot at seeing if I can fix that. Thanks for the help. – Tyler Shellberg Jul 10 '19 at 19:29
  • Yes, you need to preload it as described in the link I posted earlier. But `matlab` is a script, not an executable. You might want to make a local copy of it and include the `LD_PRELOAD` thing in that script. – Cris Luengo Jul 10 '19 at 19:30
  • @CrisLuengo So when I call ```matlab``` in a linux terminal, it's calling a script? That's where I need to put LD_PRELOAD? Where is that script located, and what is it called? – Tyler Shellberg Jul 10 '19 at 19:51
  • 1
    `which matlab` will tell you where it is. You could copy it to `matlab_asan` and modify it to add the `LD_PRELOAD` command. – Cris Luengo Jul 10 '19 at 19:52
  • @CrisLuengo It turns out that I could just call those "LD_PRELOAD" commands in the terminal rather than editing the MATLAB script. (which I tried, it was very difficult). In any case, sadly, it turned up 0 errors the moment after the actual MATLAB script started. When MATLAB was starting up there were many, but it seems the script and my MEX files are clean. – Tyler Shellberg Jul 11 '19 at 15:39
  • If you do `export LD_PRELOAD=...` in the terminal, then for **every** program started from that terminal after this library will be preloaded. Usually one does `LD_PRELOAD=... matlab`, so that the variable is only set for the one executable. But this will not work (I think) for `matlab` because it's a script. This is why I suggested you edit the script. In any case, I'm glad you got it to work. – Cris Luengo Jul 11 '19 at 15:53
  • @CrisLuengo Does that only apply for the lifetime of that terminal? In other words, if I close that terminal and launch another, LD_PRELOAD is effectively gone, isn't it? If not, then yeah, that's not pretty. – Tyler Shellberg Jul 11 '19 at 16:49
  • 1
    Yes, it's the current shell, so not terrible, just something to be aware of. If you close the terminal you exit the shell and its environment variables will be lost. – Cris Luengo Jul 11 '19 at 16:57

1 Answers1

3

Sanitized libraries (MEX-files are shared libraries) expect libasan.so to be either linked to main executable (in that case MATLAB) or LD_PRELOADed at start. As you can't rebuild MATLAB, the second approach is your only chance.

It might not work smoothly because Asan would likely find memory issues in MATLAB startup code and abort before it gets to your MEX-file. You can use export ASAN_OPTIONS=halt_on_error=false to ignore those errors (they'll still be reported but at least execution will continue).

As a side note, your issue is similar to other questions about running sanitized binary plugins in unsanitized interpreters (see e.g. similar question for Python, although that one is Clang-centric).

yugr
  • 19,769
  • 3
  • 51
  • 96
  • That question for Python seems to suggest it can be made to target just the shared library and not the executable using it. I still don't completely follow though, where exactly is LD_PRELOAD used? Is it in the ```MAKEFILE``` of my project somewhere? I found the path to GCC's libasan.so, so I'm ready once I figure out LD_PRELOAD – Tyler Shellberg Jul 10 '19 at 19:41
  • 1
    "where exactly is LD_PRELOAD used" - you need to `export LD_PRELOAD=path/to/libasan.so.N` before you run MATLAB executable. – yugr Jul 11 '19 at 04:50
  • 1
    "it can be made to target just the shared library and not the executable using it" - `libasan` works by intercepting many standard libc functions in executable and all loaded shlibs (`memset`, `malloc`, etc.). This may cause it to detect errors in these libs. I've added a hint in the answer on how to get past those errors (it's the MEX-file you are interested in after all, not other code). – yugr Jul 11 '19 at 04:53
  • Thanks for the clarification. When I attempt this, I get tons of errors after calling ```matlab``` in the terminal. I get: ```ERROR: ld.so: object '/usr/lib/gcc/x86_64-redhat-linux/8/libasan.so' from LD_PRELOAD cannot be preloaded (file too short): ignored.``` And when I run MATLAB again and try, I get the same error from before, it doesn't come first in the initial library list, you should use LD_PRELOAD. – Tyler Shellberg Jul 11 '19 at 14:48
  • Nevermind, apparently I had the path wrong. Mine is located in /lib64/libasan.so.5 It seems to be working...though it only showed errors up to the point where the MATLAB script really started. At that point, no more errors of any kind showed up. I'm not really buying that neither the matlab script nor *any* of the massive Mex file usage had any errors, but I could be wrong. – Tyler Shellberg Jul 11 '19 at 15:12
  • @TylerShellberg Congrats, that was surprisingly smooth. You can try injecting a trivial memory overflow into your Mex to check if it works (be sure to inspect generated assembler, GCC tends to optimize out really trivial overflows). – yugr Jul 11 '19 at 16:00
  • @TylerShellberg I suspect that part of the problem is that MATLAB uses it's own memory management with blackja^W GC and pools. You might succeed with manual memory poisoning (see [this post on custom allocators](https://blog.fuzzing-project.org/65-When-your-Memory-Allocator-hides-Security-Bugs.html) and [this one on manual poisoning APIs](https://github.com/google/sanitizers/wiki/AddressSanitizerManualPoisoning)). – yugr Jul 11 '19 at 16:04
  • 1
    @TylerShellberg And another guess is that MATLAB redirects stderr from Mex (in that case `export ASAN_OPTIONS=log_path=XXX` would help). – yugr Jul 11 '19 at 16:07
  • When I attempt to use ```export ASAN_OPTIONS``` the result is "We could not determine the path of the MATLAB root directory". Very odd. – Tyler Shellberg Jul 11 '19 at 18:01
  • 1
    @TylerShellberg You'll need to combine both options: `export ASAN_OPTIONS=halt_on_error=false:log_path=...` – yugr Jul 11 '19 at 18:06
  • That seems to still cause the problem. I'm just setting the log_path to the ./asanlog.txt. – Tyler Shellberg Jul 11 '19 at 18:12
  • @TylerShellberg The error message says that `matlab` wrapper script failed to locate the MATLAB proper (it does this via some awk magic). I suspect there's a problem with using `.` in log_path, try using absolute dir. If that does not help you'll have to debug the script (e.g. via `sh -x`). – yugr Jul 11 '19 at 18:28
  • Tested with absolute path, it works perfect now. However, the resulting logs are the same, it seems to stop once the actual MATLAB script starts. So either something's wrong, or there's no issue in the script or my code. – Tyler Shellberg Jul 12 '19 at 14:47
  • @TylerShellberg I'd try inserting deliberate memory error into your code (with caveats outlined above). If it's detected, your unchanged code is indeed ok. – yugr Jul 12 '19 at 14:59
  • Thanks, I'll try that. As a sidenote, I'm having a weird issue. I removed the flags from my makefile for ```fsanitize=address```, make clean, make and replaced the old files...but MATLAB is now warning me about an undefined symbol ```asan_option_detect_stack_use_after_return```. Why? I completely removed any mention of ASAN from my makefile and rebuilt. :( – Tyler Shellberg Jul 12 '19 at 15:29
  • @TylerShellberg Looks like something somewhere hasn't been recompiled or reinstalled... – yugr Jul 12 '19 at 16:23
  • By the way, when I try to use @ in the comments, it just isn't working for me. Sometimes it does, but rarely. As for the rest, I don't know what to do. I think I somehow permanently broke my MATLAB, or maybe something in the OS itself. I've quadruple checked and re-built it and re-moved the files a dozen times, nothing works. I checked the environment variables, there's nothing odd there. – Tyler Shellberg Jul 12 '19 at 16:26
  • I completely uninstalled libasan. Somehow, it's still showing up in the symbol table of my built mex files. I hate this. – Tyler Shellberg Jul 12 '19 at 16:48
  • @TylerShellberg I'm afraid I don't have an explanation. There's nothing magical about `-fsanitize=address`, it just links shlib with a runtime library. I can't imagine how it could damage your MATLAB installation... – yugr Jul 12 '19 at 16:59
  • "By the way, when I try to use @ in the comments" - SO will not let you use it when it's redundant. – yugr Jul 12 '19 at 17:01
  • Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/196387/discussion-between-yugr-and-tyler-shellberg). – yugr Jul 12 '19 at 17:01