dll export function pointer (best way to provide exported function hooking)

Question

I am trying to export a function pointer for a function to be called. What I am after is when a function in a dll/exe needs to call a function exported by another library it gets the function pointer and calls that. The reason for this is I want to provide a hooking mechanism and I figured function pointers would be the quickest and easiest way because I can change what they point to easily are runtime. So I found this Exporting a function pointer from dll and I cant get it to work. Whenever I call it to get the function pointer I get an error that it cant find the entry point. So the error isnt that the function pointer is working but the function to get the function pointer isnt working. I believe it is a function signature issue. Here is an example:

Colors.h

#ifndef __COLORS
#define __COLORS

#ifdef  MYDLL_EXPORTS 
/*Enabled as "export" while compiling the dll project*/
#define DLLEXPORT __declspec(dllexport)  
#else
/*Enabled as "import" in the Client side for using already created dll file*/
#define DLLEXPORT __declspec(dllimport)  
#endif

#include <string>
#include <vector>
class Colors
{
private:
     std::string myColor;
     static DLLEXPORT std::vector<std::string> allColors;
public:
     Colors(){};
     Colors(std::string MyColor);
     virtual DLLEXPORT std::string getMyColor();
     virtual DLLEXPORT void addToColors(std::string color);
     std::vector<std::string> getAllColors();
};
typedef Colors* (*create)(std::string);

DLLEXPORT create createColors();

Colors* createColors2(std::string color);



#endif

colors.cpp

#define MYDLL_EXPORTS
#include "Color.h"




std::vector<std::string> Colors::allColors;

Colors::Colors(std::string MyColor)
{
     this->myColor = MyColor;
     this->allColors.push_back(this->myColor);
}

std::vector<std::string> Colors::getAllColors()
{
     return this->allColors;
}

std::string Colors::getMyColor()
{
    return this->myColor;
}

Colors* createColors2(std::string color)
{
    return new Colors(color);

}

DLLEXPORT void Colors::addToColors(std::string color)
{
     this->allColors.push_back(color);
}

DLLEXPORT create createColors()
{
     return &createColors2;
}

main.cpp

#define MYDLL_EXPORTS

#include <iostream>
#include <Windows.h>
#include "Color.h"



int main()
{


    Colors red("red");
    Colors blue("blue");

    Colors* dlltest;

    //Define the function prototype
    typedef Colors* (*createNewColor)();

    BOOL freeResult, runTimeLinkSuccess = FALSE;
    HINSTANCE dllHandle = NULL;
    createNewColor dllCreateNewColor = NULL;

    //Load the dll and keep the handle to it
    dllHandle = LoadLibrary(L"libs/testerdll.dll");

    // If the handle is valid, try to get the function address. 
    if (NULL != dllHandle)
    {

        //Get pointer to our function using GetProcAddress:
        dllCreateNewColor = (createNewColor)GetProcAddress(dllHandle,"createNewColor");

        // If the function address is valid, call the function. 
        if (runTimeLinkSuccess = (NULL != dllCreateNewColor))
        {

            dlltest = dllCreateNewColor();
            std::cout << "Color of dll class: " << dlltest->getMyColor() << std::endl;
        }
        else
        {
            std::cout << "Failed to locate function" << std::endl;
        }

            //Free the library:
            //freeResult = FreeLibrary(dllHandle);
    }
    else
    {
         std::cout << "Failed to load library" << std::endl;
    }

    std::vector<std::string> colorslist = red.getAllColors();
    for (std::string color : colorslist)
    {
        std::cout << color << std::endl;
    }
    return 0;
}

Dll project dllmain.cpp

// testerdll.cpp : Defines the exported functions for the DLL application.


#include "stdafx.h"


#include "Color.h"



__declspec(dllexport) Colors* createNewColor()
{
    create temp1 = createColors();  //seems to fail here

    return nullptr;
}

Yes I know I have memory leaks etc. this was just a quick example code to replicate the problem.

There's no need to `DLLEXPORT` virtual functions -- they are already found via function pointers in the vtable :) — Ben Voigt, Oct 27 '15 at 19:16
ben and Dieter can you please provide examples? Im trying to learn dll files — lesyriad, Oct 27 '15 at 19:25

score 0 · Answer 1 · edited May 23 '17 at 11:44

To return the function, you need to get it's address, and return that

e.g.

__declspec(dllexport) create createNewColor()
{
    create temp1 = createColors;

    return temp1;
}

However, this system (using std::string as a return type, requires that both the .exe and the .dll use the same DLL based runtime library.

stackoverflow : passing reference to STL over function boundary

C++ does not define a calling convention between files. This means that different compilers may set up C++ objects slightly differently. Microsoft limited that with the definition of COM, but that still is a possibility.

Also for visual studio, there are separate heaps (new / delete) between runtime instances. When you link against the dynamic library, all dlls and exes in the process share this DLL. But then they all need to be updated together.

So this process can work, but be wary about :-

Sharing C++ types between binaries (DLL/EXE) - no ABI
Using new in DLL, and delete in EXE. (different heaps).

STL objects are also problematic, as they are a mixture of header implementation (compiled into the binary), and DLL implementation (compiled into C++ runtime).

okay thanks got it to work. is there a better way to implement a function hooking api? The program as a whole want to load dll's/plugins and if one of them needs to hook a specific function on the main exe so future calls from other plugins execute a function inside of it instead (hooked function) is there a good proper way to do that? I just thought this would be the best way via function pointers. — lesyriad, Oct 27 '15 at 21:42

dll export function pointer (best way to provide exported function hooking)

1 Answers1