Out-Of-Proc-COM-Server: BSTR not properly marshalled

Question

I have developed a Out-Of-Proc-COM-Server in C++ with Visual Studio 2010 to avoid the 64-bit vs. 32-bit problem in Shellextensions (http://blog.mattmags.com/2007/06/30/accessing-32-bit-dlls-from-64-bit-code/).

I described the Interfaces like here (http://msdn.microsoft.com/en-us/library/ms686605%28v=VS.85%29.aspx) in a IDL-File:

import "unknwn.idl";
[
 object,
 uuid("xx"),
 helpstring("IShellServerx86-Interface")
]
interface IShellServerx86 : IUnknown 
{
   HRESULT ShowFileInfo([in]BSTR file, [out]BSTR* htmlFile, [in]BSTR pathChar);
};

This file generates me a Proxy/Stub-DLL which I also registered to use the Standard Marshaller methods. If I call now

IShellServerx86* pShellServer = NULL;
CoCreateInstance(__uuidof(CShellServerx86), NULL, CLSCTX_LOCAL_SERVER,
                 __uuidof(IShellServerx86), (void**)&pShellServer);

the server is created and I can call the method

HRESULT CShellServerx86::ShowFileInfo(BSTR file, BSTR* htmlFile, BSTR pathChar)

and with the created parameters (client-side):

BSTR filebstr = ::SysAllocString(A2OLE(file));
BSTR pathBstr = ::SysAllocString(A2OLE(pathChar));
BSTR htmlFileBstr = ::SysAllocString(A2OLE(""));

In the client the BSTR's are correctly generated but when the COM-method is called (he finds it!) and I debug into the dllhost.exe, the parameters are invalid like the wrong encoding is chosen. I tried for whole project to set "Unicode" but nothing changes.

Have I forgotten any settings or should I try other data types for marshalling?

Thank you for help in advance.

EDIT:

The implementation of the client is:

int CShellWrapperx64Module::ShowFileInfo(IN const char* file, 
                                                    OUT VARIANT &htmlFile,
                                                    IN const char* pathChar)
{...
    ::CoInitialize(NULL);
    IShellServerx86* pShellServer = NULL
    hr = ::CoCreateInstance(__uuidof(CShellServerx86), NULL, 
                           CLSCTX_LOCAL_SERVER, __uuidof(IShellServerx86),
                           (void**)&pShellServer);
    BSTR filebstr = ::SysAllocString(A2OLE(file));
    BSTR pathBstr = ::SysAllocString(A2OLE(pathChar));
    BSTR htmlFileBstr = ::SysAllocString(A2OLE(""));
    //Call method of Server
    hr = pShellServer->ShowFileInfo(filebstr, &htmlFileBstr, pathBstr);
    ::CoUninitialize();
    VariantInit(&htmlFile);
    htmlFile.vt = VT_BSTR;
    htmlFile.bstrVal = htmlFileBstr;
}

The server method is declared as following:

HRESULT CShellServerx86::ShowFileInfo(BSTR file, BSTR* htmlFile, BSTR pathBSTR)
{...
 //TODO
}

In the server and client methods the debugger recognize the BSTR-strings as wchar_t*-arrays. But the content for example for the string "file" in the server method is something like: 0x02546e80 "㤈榧".

The encoding is for all projects (client/server) set to Multibyte-Encoding (Visual Studio).

EDIT2:

The server is declared as follwed:

class IShellServerx86 : public IUnknown {
  public:

  virtual HRESULT ShowFileInfo(BSTR file, BSTR* htmlFile, BSTR pathChar) = 0;

};

Implementation of the interface:

//CoClass from Interface (Implementation)
class CShellServerx86 : public IShellServerx86 {
 public:
  CShellServerx86();
  virtual ~CShellServerx86();
  //inherited from IUnknown
  ULONG STDMETHODCALLTYPE AddRef(void);
  ULONG STDMETHODCALLTYPE Release(void);
  HRESULT STDMETHODCALLTYPE QueryInterface(REFIID riid, void** ppv);

  HRESULT ShowFileInfo(BSTR file, BSTR* htmlFile, BSTR pathChar);

 protected:
  ULONG m_uRefCount;
};

... and class-factory class CShellServerx86ClassFactory : public IClassFactory { public: CShellServerx86ClassFactory(); ~CShellServerx86ClassFactory();

 //inherited methods from IUnknown
 ULONG STDMETHODCALLTYPE AddRef(void);
 ULONG STDMETHODCALLTYPE Release(void);
 HRESULT STDMETHODCALLTYPE QueryInterface(REFIID riid, void** ppv);

 //inherited methods from IClassFactory
 HRESULT STDMETHODCALLTYPE CreateInstance(IUnknown *pUnkOuter, 
                                          REFIID riid, void** ppv);
 HRESULT STDMETHODCALLTYPE LockServer(BOOL fLock);

protected: ULONG m_uRefCount; };

GetClass-Method from the DLL:

STDAPI DllGetClassObject ( REFCLSID rclsid, REFIID riid, void** ppv ) {
  if (!::InlineIsEqualGUID(rclsid, __uuidof(CShellServerx86)) ) {
     return CLASS_E_CLASSNOTAVAILABLE;
  }
  *ppv = NULL;
  CShellServerx86ClassFactory* pShellServerFac;
  pShellServerFac = new CShellServerx86ClassFactory;
  if (pShellServerFac == NULL) {
     return E_OUTOFMEMORY;
  }
  pShellServerFac->AddRef();
  HRESULT hr = pShellServerFac->QueryInterface(riid, ppv);
  pShellServerFac->Release();
  return hr;

}

Answer 1

First you have to inspect what A2OLE produces in your case and whether that's suitable input for SysAllocString().

Then you have to implement that //TODO - it's the callee responsibility to properly build values of out parameters. You'll have to do something like this:

HRESULT CShellServerx86::ShowFileInfo(BSTR file, BSTR* htmlFile, BSTR pathBSTR)
{
    if( htmlFile == 0 ) {
       return E_POINTER;
    }
    // do useful stuff, generate the string for the htmlFile, then
    *htmlFile = SysAllocString( TheStringForHtmlFileParameter );
    return S_OK;
}

Also you're leaking a BSTR in the caller:

BSTR htmlFileBstr = ::SysAllocString(A2OLE(""));
//Call method of Server
hr = pShellServer->ShowFileInfo(filebstr, &htmlFileBstr, pathBstr);

will lose the BSTR passed as second parameter since a new BSTR will be created by the callee. Instead just initialize it to a null pointer:

BSTR htmlFileBstr = 0;
//Call method of Server
hr = pShellServer->ShowFileInfo(filebstr, &htmlFileBstr, pathBstr);

Also you leak all BSTRs anyway since you don't call SysFreeString() when done. Either call SysFreeString() on each BSTR you own or better use a wrapper class like ATL::CComBSTR or _bstr_t.