How to return value from DLL using parameter of function as a pointer in C++?

Question

I have a simple DLL:

dllmain.cpp:

#define MYDLLDIR        
#include "pch.h"

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    ...
}


void callByPtr(int *i) {
    
    (*i)++;
}

pch.h

#include "framework.h"

#ifdef MYDLLDIR
#define DLLDIR __declspec(dllexport) __stdcall
#else
#define DLLDIR __declspec(dllimport)
#endif

extern "C" {

    DLLDIR void callByPtr(int *i);
        
};

Client:

typedef void(__stdcall* callByPtr)(int*);

int main()
{
    HINSTANCE hDLL;

    hDLL = LoadLibrary(_T("MyDll.dll"));

    if (NULL != hDLL)
    {

        callByPtr myCall = (callByPtr)GetProcAddress(hDLL, "callByPtr");

        if (!myCall) {
            return EXIT_FAILURE;
        }

        int i = 10;

        int* ptri = &i;

        std::cout << "i " << i << std::endl;
        std::cout << "ptri " << ptri << std::endl;

        myCall(ptri);

        std::cout << "---- After Call ----\n";

        std::cout << "i " << i << std::endl;
        std::cout << "ptri " << ptri << std::endl;    

    }
}

Result:

---- Before Call ----

i = 10

ptri = 0025FB40

---- After Call ----

i = 11286192

ptri = 0025FB3C

The adress of ptri has changed and value was not 11.

How to implement this properly that I can get a value from DLL using method above?

Thank you!

StPiere · Accepted Answer · 2020-09-18T10:23:10.463

1

Your exporting definitions are also not correct. Should be something like:

#ifdef MYDLL_EXPORT
#define MYDLLDIR  __declspec(dllexport)
#else
#define MYDLLDIR __declspec(dllimport)
#endif

and use the same macro (MYDLLDIR) for both export (dll, #MYDLL_EXPORT defined) and import(clients, #MYDLL_EXPORT NOT defined)

You have to use the same calling convention for callByPtr in all places, in your case __stdcall (the default one is __cstdcall).

In your pch.h then:

MYDLLDIR void __stdcall callByPtr(int *i);

edited Sep 18 '20 at 10:23

answered Sep 18 '20 at 07:34

StPiere

4,113
15
24

But I'm not using the "import" – kov_george Sep 18 '20 at 07:36
you do use DLLDIR void callByPtr(int *i); in your pch.h. This is not correct. See my edit – StPiere Sep 18 '20 at 07:44
I'm so sorry it was my typo, now I edited it but it does not help – kov_george Sep 18 '20 at 07:53
as suggested by other user -> remove __stdcall completely or place it after return type – StPiere Sep 18 '20 at 10:24

score 0 · Answer 2 · answered Sep 18 '20 at 09:17

0

since you return void in your exported function DLLDIR void callByPtr(int *i); you should use the default calling convention for C and C++ programs __cdecl.
After changing:

in your pch.h file:
#define DLLDIR __declspec(dllexport) __stdcall
to
#define DLLDIR __declspec(dllexport)
in your Client file:
typedef void(__stdcall* callByPtr)(int*);
to
typedef void(__cdecl* callByPtr)(int*);

The rebuild is without errors and warnings and the output is like that:

i 10
ptri 0113FCA4
---- After Call ----
i 11
ptri 0113FCA4

answered Sep 18 '20 at 09:17

Petr Dokoupil

376
4
9

What if I want to return let say "int" shall I use __stdcall? – kov_george Sep 18 '20 at 09:38
No, sorry to put it wrong, you can also use __cdecl, or as @CristiFati mentioned use the default. – Petr Dokoupil Sep 18 '20 at 15:13

CristiFati · Answer 3 · 2020-09-18T22:31:55.023

According to [MS.Docs]: __stdcall

Syntax

return-type __stdcall function-name[( argument-list )]

the calling convention specifier comes after the function return type. The way you defined it is before, so (probably) the compiler ignored it^???, ending up in the .dll exporting the function as __cdecl (default), and when the .exe called it as __stdcall, Bang! -> Stack Corruption, and what you think is your pointer is actually something completely different, hence your weird outputs.
The thing that is interesting is that on my end, the compiler (VS2017) spits error C2062: type 'void' unexpected when I try to build the .dll using your form (#define DLL00_EXPORT_API __declspec(dllexport) __stdcall).

Below is a working example (I modified the file names and contents).

dll00.h:

#pragma once

#if defined(_WIN32)
#  if defined(DLL00_EXPORTS)
#    define DLL00_EXPORT_API __declspec(dllexport)
#  else
#    define DLL00_EXPORT_API __declspec(dllimport)
#  endif
#else
#  define DLL00_EXPORT_API
#endif

#if defined(CALL_CONV_STDCALL)
#  define CALL_CONV __stdcall
#else
#  define CALL_CONV
#endif

#if defined(__cplusplus)
extern "C" {
#endif

DLL00_EXPORT_API void CALL_CONV callByPtr(int *pI);

#if defined(__cplusplus)
}
#endif

dll00.cpp:

#define DLL00_EXPORTS
#include "dll00.h"


void CALL_CONV callByPtr(int *pI) {
    if (pI) {
        (*pI)++;
    }
}

main00.cpp:

#include <iostream>
#include <Windows.h>
#include "dll00.h"

#if defined(CALL_CONV_STDCALL)
#  define FUNC_NAME "_callByPtr@4"
#else
#  define FUNC_NAME "callByPtr"
#endif

using std::cout;
using std::endl;


typedef void(CALL_CONV *CallByPtrFunc)(int*);


int main() {
    HMODULE hDLL;

    hDLL = LoadLibrary("dll00.dll");

    if (!hDLL) {
        std::cout << "LoadLibrary failed" << std::endl;
        return -1;
    }

    CallByPtrFunc callByPtr = (CallByPtrFunc)GetProcAddress(hDLL, FUNC_NAME);

    if (!callByPtr) {
        std::cout << "GetProcAddress failed" << std::endl;
        CloseHandle(hDLL);
        return EXIT_FAILURE;
    }

    int i = 10;

    int *ptri = &i;

    std::cout << "i " << i << std::endl;
    std::cout << "ptri " << ptri << std::endl;

    callByPtr(ptri);

    std::cout << "---- After Call ----\n";

    std::cout << "i " << i << std::endl;
    std::cout << "ptri " << ptri << std::endl;

    CloseHandle(hDLL);

    return 0;
}

Output:

[cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q063951075]> sopr.bat
*** Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ***

[prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2017\VC\Auxiliary\Build\vcvarsall.bat" x86
**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.27
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x86'

[prompt]>
[prompt]> dir /b
dll00.cpp
dll00.h
main00.cpp

[prompt]> :: Build the .dll (passing /DCALL_CONV_STDCALL)
[prompt]> cl /nologo /MD /DDLL /DCALL_CONV_STDCALL dll00.cpp  /link /NOLOGO /DLL /OUT:dll00.dll
dll00.cpp
   Creating library dll00.lib and object dll00.exp

[prompt]>
[prompt]> :: Build the .exe (also passing /DCALL_CONV_STDCALL)
[prompt]> cl /nologo /MD /W0 /EHsc /DCALL_CONV_STDCALL main00.cpp  /link /NOLOGO /OUT:main00.exe
main00.cpp

[prompt]>
[prompt]> dir /b
dll00.cpp
dll00.dll
dll00.exp
dll00.h
dll00.lib
dll00.obj
main00.cpp
main00.exe
main00.obj

[prompt]>
[prompt]> main00.exe
i 10
ptri 00F5FCC8
---- After Call ----
i 11
ptri 00F5FCC8

[prompt]> :: It worked !!!
[prompt]>
[prompt]> dumpbin /EXPORTS dll00.dll
Microsoft (R) COFF/PE Dumper Version 14.16.27043.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file dll00.dll

File Type: DLL

  Section contains the following exports for dll00.dll

    00000000 characteristics
    FFFFFFFF time date stamp
        0.00 version
           1 ordinal base
           1 number of functions
           1 number of names

    ordinal hint RVA      name

          1    0 00001000 _callByPtr@4

  Summary

        1000 .data
        1000 .rdata
        1000 .reloc
        1000 .text

Notes:

Obviously, the code presented is not the one generating your output:
- It doesn't compile:
  - The error I mentioned at the beginning (although that could probably be avoided using some older compiler (or (doubtfully?) a compiler flag))
  - Missing #includes and usings in your .exe code
"Minor" code problems:
- NULL pointer test before dereferencing
- CloseHandle
- Differences between DLLDIR definition (as also mentioned by @Petr_Dokoupil): __declspec(dllexport) __stdcall vs. __declspec(dllimport), but since you're dynamically loading the .dll instead of linking to it, it has nothing to do with the error
Why the need to use __stdcall? (answer provided in comment: "to be compatible with other languages"):
- It only matters on 32bit (on 64bit it's ignored)
- It introduces a lot of additional issues (some of them you didn't even experienced), like the function name mangling (check dumpbin output), which can only be avoided using a .def file
- All in all, this seems like an XY Problem. You should use defaults (get rid of __stdcall completely):
  - With this version of the code, simply don't pass /DCALL_CONV_STDCALL argument to the compiler when building the .dll and .exe
  - You're less likely to run into (subtle) problems (at least until you get more experience in this area)
  - Removing all the calling convention related code, would make it much shorter and cleaner
- All the listed languages (Delphi, Python, C#, ...) have support for __cdecl (after all, I think that most of machine code running out there, is still written in C)

For more details regarding this whole area, you could check (including (recursively) referenced URLs):

Very useful notes, thank you! I was considering to use __stdcall to be compatible with other languages. — kov_george, Sep 18 '20 at 11:02
What other languages? In *C#* for example, you can specify the (unmanaged function) calling convention. Anyway, as I shown, it **is** possible to use *\_\_stdcall*, but it's a bit more tricky. — CristiFati, Sep 18 '20 at 11:11

How to return value from DLL using parameter of function as a pointer in C++?

3 Answers3

Syntax