Compiler keeps inlining functions and array values

Question

I am attempting to implement an anti-analysis technique where some key functions are encrypted and only decrypted before usage and then encrypted again. So my implementation involves having an array which will hold the function addresses and length which the external crypter will use to identify and encrypt said functions.

My problem is the compiler keeps using the values in the array as constants instead of accessing the which makes the crypter unable to located the functions and so on, I've tried several compiler options but none does what I am looking for. I am looking for the way to make it so the function array to have a single copy and not be inlined because currently.

Edit: Compiler is VS2019 Edit 2: Clarifying the problem

Which compiler/toolchain are you using? This sounds like you'll need to disable optimization in places, and not use Link Time Code Generation. — 1201ProgramAlarm, Mar 11 '20 at 15:44
Is this a toy project for research, recreational, or educational purposes or are you expecting this to withstand an actual attack? — David Schwartz, Mar 11 '20 at 15:47
I'm not quite sure I fully understand the behavior here. You have some functions which you have encrypted in some way. And to call them at runtime, you need to decrypt the functions, at which point you will invoke the decrypted routine and then remove the decrypted versions of the function. If this is the case, are you trying to prevent inlining on the routine that decrypts the encrypted functions, or are you trying to prevent inlining on the encrypted functions themselves? If it's the former, I don't know why it matters, and if it's the latter, how is that even possible? — Nicol Bolas, Mar 11 '20 at 15:49
Also, are you starting your executable with these functions *not* encrypted? Because if you are, then you've already failed at defending against whatever you are defending against. — Nicol Bolas, Mar 11 '20 at 15:50
If you question is "How to stop the compiler from inlining", then this is a dupe of: https://stackoverflow.com/questions/3329214/is-it-possible-to-force-a-function-not-to-be-inlined — NathanOliver, Mar 11 '20 at 15:51
@NicolBolas yes, the reason is it either uses a duplicate array or uses constants instead which makes the crypter unable to locate and encrypt the function in the place, since it uses the array to identify the functions and their sizes. — Diab, Mar 11 '20 at 16:49
@DavidSchwartz it's mainly for research but coupled with different other methods, should be annoying enough to thwart some attackers. — Diab, Mar 11 '20 at 16:51

score 1 · Accepted Answer · answered Mar 12 '20 at 14:51

The array needs to be exported from the binary, i.e. it needs to be a public symbol in the executable's symbol table.

On Windows, that would be:

using Function = ...;

__declspec(dllexport) const Function encrypted_functions[] = { f1, f2, ... };

On Non-Windows platforms, you need to use the gold linker and ensure the symbol is visible:

#ifdef __cplusplus
extern "C" {
#endif
__attribute__((visibility ("default")) const Functions encrypted_functions[] = ...;
#ifdef __cplusplus
}
#endif

And then use the gold linker's --export-dynamic-symbol=encrypted_functions option to add the symbol to the export table (or equivalent in the LLVM linker). The name may be mangled even though it's a C symbol, so you'd need to use objdump to examine the object file to see what's the real symbol name of that array.

But it's all a bit silly, since the encryption program should be a part of the build process and should interact with the object files directly. The best way would be using libObject bundled with the LLVM project. See the source files, the headers, and some documentation.

Compiler keeps inlining functions and array values

1 Answers1