Preventing Visual C++ From Inlining a Function

Question

i wrote a simple program in c++ :

#include <stdio.h>

const signed char pass[] = "\x70\x61\x73\x73\x77\x6F\x72\x64";

bool __stdcall check_password(const signed char * str)
{
    unsigned int i;
    for(i=0;i<8;++i)
        if(str[i]!=pass[i])
            return false;
    return true;
}

int main(int argc, char * argv[])
{
    signed char buf[20];
    printf("please enter the password: ");
    scanf("%s",buf);
    printf((check_password(buf)) ? "correct!\n" : "incorrect.\nPress any key to exit..\n");
    getchar();
    return 0;
}

and compiled it with visual studio express 2010.
i opened the result in OllyDbg, and this is what i saw:

Address   Hex dump            Command                                          Comments
00FF1011  ³.  8B35 A020FF00   MOV ESI,DWORD PTR DS:[<&MSVCR100.printf>]
00FF1017  ³.  68 0021FF00     PUSH OFFSET pass2.00FF2100                       ; ASCII "please enter the password: "
00FF101C  ³.  FFD6            CALL ESI
00FF101E  ³.  8D45 E8         LEA EAX,[LOCAL.6]
00FF1021  ³.  50              PUSH EAX
00FF1022  ³.  68 1C21FF00     PUSH OFFSET pass2.00FF211C                       ; ASCII "%s"
00FF1027  ³.  FF15 A820FF00   CALL DWORD PTR DS:[<&MSVCR100.scanf>]
00FF102D  ³.  83C4 0C         ADD ESP,0C
00FF1030  ³.  33C0            XOR EAX,EAX
00FF1032  ³>  8A4C05 E8       MOV CL,BYTE PTR SS:[EAX+EBP-18]
00FF1036  ³.  3A88 F420FF00   CMP CL,BYTE PTR DS:[EAX+pass2.pass]             ; ASCII "password"
00FF103C  ³. 75 0D           JNE SHORT pass2.00FF104B
00FF103E  ³.  40              INC EAX
00FF103F  ³.  83F8 08         CMP EAX,8
00FF1042  ³. 72 EE           JB SHORT pass2.00FF1032
00FF1044  ³.  B8 2021FF00     MOV EAX,OFFSET pass2.00FF2120                    ; ASCII "correct!\n"
00FF1049  ³. EB 05           JMP SHORT pass2.00FF1050
00FF104B  ³>  B8 2C21FF00     MOV EAX,OFFSET pass2.00FF212C                    ; ASCII "incorrect.\nPress any key to exit..\n"
00FF1050  ³>  50              PUSH EAX
00FF1051  ³.  FFD6            CALL ESI
00FF1053  ³.  83C4 04         ADD ESP,4
00FF1056  ³.  FF15 9C20FF00   CALL DWORD PTR DS:[<&MSVCR100.getchar>]

And it looks like the function check_password has been inlined, because I can't see any CALL instruction to it.

So my question is, how can I prevent this from happening? is there any way to tell the compiler that I want the function to appear in the executable?

The reason I'm asking is because I'm trying to perform DLL injection, and my primary goal is to redirect the call to check_password into another function.

There is no such thing as a macro in asm, what your referring to is the function has been inlined, usually when the compiler decides to do it it's a good thing, check this out http://stackoverflow.com/questions/3329214/is-it-possible-to-force-a-function-not-to-be-inlined if u still wanna prevent the behavior — aaronman, Jul 30 '13 at 23:25
adding "__declspec(noinline)" solved the problem. thank you! — hcf, Jul 30 '13 at 23:31
I have to say that "__declspec()" sounds awfully non-portable to me. I would think that putting the function into its own source file would be sufficient to make sure it's not inlined. I liked aaronman's suggestion to call it through a pointer. — Edward Falk, Jul 30 '13 at 23:52
@EdwardFalk I suggested that originally but I did some more research and compilers can still sometimes inline function pointers — aaronman, Jul 30 '13 at 23:54
@EdwardFalk: Modern linkers can and do inline functions in other translation units. But if you take the address of a function and do something with it, that's usually enough to coax the linker to leave a standalone callable version somewhere at least, even if it doesn't actually use that copy. In VC you can also mark the function as `__declspec(dllexport)`, which makes the function callable from a dll (or vice versa) — Mooing Duck, Jul 30 '13 at 23:54
@EdwardFalk now my answer contains solutions for the top three (I think) c++ compilers — aaronman, Jul 31 '13 at 00:16

score 5 · Accepted Answer · edited May 23 '17 at 11:50

There is no such thing as a macro in asm(it is just text replacement your code and takes place before the code is compiled). What your referring to is the function has been inlined, usually when the compiler decides to do it it's a good thing, there is no c++ wide way to do this but you can do something like this

class X {
     __declspec(noinline) int member_func() {
          return 0; 
     }
};

code taken from this answer.
This will prevent the function from being inlined. Keep in mind that most of the time inlining is a good thing because it gets rid of the overhead of the function call, it also has no real negative affects other than code bloat and debugging. Compilers are very smart and generally make good decisions about inlining. As you may already know you can use the inline specifier in c++ to suggest to the compiler that the function should be inlined.

I also originally thought that a more portable way to prevent function inlining would be to call it from a function pointer, however this post says otherwise.

For people looking at this answer using gcc you can prevent inline like this void void __attribute__ ((noinline)) myfunc() this looks like it works for clang too ( I am using Apple clang version 4.0 ) so that should cover most of the c++ compilers people will be using.

Preventing Visual C++ From Inlining a Function

1 Answers1