0

I have a native dll (bar.dll) that does almost nothing:

It exports one function foo that opens calc.exe.

And it does the same on DLL_PROCESS_ATTACH.

I've read here that once everything else fails

The directories that are listed in the PATH environment variable

are searched.

I've placed bar.dll in a folder that's in PATH (and when I use the where it's located successfully).

But when I try to run

rundll32 bar.dll,foo

It fails with

There was a problem starting bar.dll

bar.dll is not a valid Win32 application.

(I've tried compiling it both as 32 and 64 bit) It does work if I'm in the same directory.

The source code

bar.cpp

#include "stdafx.h"
#include "bar.h"
#include <iostream>

BAR_API void __stdcall foo(HWND hwnd, HINSTANCE hinst, LPSTR lpszCmdLine, int nCmdShow)
{
    system("calc");
}

bar.h

#ifdef BAR_EXPORTS
#define BAR_API __declspec(dllexport)
#else
#define BAR_API __declspec(dllimport)
#endif

#pragma comment(linker, "/EXPORT:foo=_foo@0")

extern "C" {    
    BAR_API void __stdcall foo(HWND hwnd, HINSTANCE hinst, LPSTR lpszCmdLine, int nCmdShow);
}

dllmain.cpp

#include "stdafx.h"
#include "bar.h"
#include <iostream>

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
                     )
{
    switch (ul_reason_for_call)
    {
    case DLL_PROCESS_ATTACH:
    case DLL_THREAD_ATTACH:
    case DLL_THREAD_DETACH:
    case DLL_PROCESS_DETACH:
        break;
    }
    return TRUE;
}
Rob B
  • 1
  • 2
  • 1
    Apparently relevant: [Why CreateProcess must not be called from a DllMain function?](http://stackoverflow.com/q/26754014/11683) – GSerg May 01 '16 at 16:24
  • Didn't even use CreateProcess, just `system("calc")` :) – Rob B May 01 '16 at 16:31
  • 1
    @RobB: Please read the source code for `system`. It **will** eventually call into `CreateProcess`, but performs **lots** of other dangerous stuff, that definitely should not go into your `DllMain` function. – IInspectable May 01 '16 at 16:51
  • Rob, could you please clarify whether the call to system() in DllMain() or in foo()? Do you have a DllMain() function at all? Have you tried using Process Monitor (available from the MS web site) to see exactly where rundll32 is looking? Have you read all the way through https://support.microsoft.com/en-us/kb/164787 ? – Harry Johnston May 02 '16 at 00:17
  • 1
    On second thoughts: on my machine I get "The specified module could not be found" if rundll32 can't find the DLL. Perhaps the DLL you're loading depends on another DLL, and it's the other DLL that can't be found? (If in doubt, Process Monitor will show you what's going on.) – Harry Johnston May 02 '16 at 00:21
  • @HarryJohnston I've attached the code to clarify – Rob B May 02 '16 at 05:41
  • Why on earth are you using rundll32 with a function signature that doesn't conform to [the expected one](https://support.microsoft.com/en-us/kb/164787)? Actually, why are you using rundll32 at all? – Matteo Italia May 02 '16 at 05:46
  • @MatteoItalia 1. Because it generally does seem to work and it's just supposed to check something not to serve as a product, from a practical point of few doesn't it just result in junk being left on the stack frame ? (Of course it's not something I'd want but it shouldn't prevent this from running) 2. Because I want to specifically check something regarding rundll32. FWIW it changing to a conforming signature had no effect. – Rob B May 02 '16 at 06:05
  • I don't think this is your problem, but you don't define BAR_EXPORTS so you're not declaring the foo() function properly. Also `dllmain.cpp` is expecting foo() to be in a DLL rather than statically linked, though *perhaps* that doesn't matter since it doesn't use it. Note that you don't need an empty DllMain, you can just leave it out. What does Process Monitor show? – Harry Johnston May 02 '16 at 21:35

0 Answers0