I don't know what your particular problem was, but I was also having a problem using WCF services within the same process. My problem was that my application would hang for a minute upon calling the service from the client, and finally return with a timeout exception. I suppose this has something to do with a blocking call on the thread while waiting for a response from the server... which is a problem if the server is being blocked. I was using named pipes, I didn't try any of the other options.
Samson's suggestion above probably works, but I did it the other way. I created the service and listener via a thread instead of calling the hosted service on a thread. This seems to be working just fine.
Now for something a little off topic - Firestrand asked why you would want to do this from within your own application. I am creating a Windows service, and a separate WPF application UI for which it needs to communicate. However, debugging Windows services are kind of a pain, so much of the service work, including WCF calls, is going into a separate DLL which I can reference in the UI application while in "test" mode, which essentially just calls the startup routines that the actual Windows service would. This allows me to debug both the UI and the "simulated" service without attaching to processes, running multiple processes, etc. It seems to be working great. Once the UI and DLL are completed I will simply create the actual Windows service which is basically just a wrapper for the DLL, and then at least hide the test mode on the UI. Just thought I'd share in case someone else finds this approach may be helpful.