11

How can I stream data (text) from a managed assembly to a native library and stream data (text) back to the managed assembly?

Specifically, I want to expose a System.IO.Stream of some sort on the .NET side, and (most importantly) a FILE * on the native side.

The signature of the native method should be:

FILE * foo(FILE * bar);

The signature of a wrapper around the native p/invoke call should be:

CustomStream foo(CustomStream bar);

I do not want to use callback methods on the native side (one for getting more data and one for setting more data). I want to use a FILE * on the native side - and all of the associated methods which operate upon it such as fprintf.

I do not want any disk I/O. This needs to be an in-memory operation.

I have complete control over both the managed assembly and the native library.

The solution must work with .NET 2.0

I'm willing to create any sort of managed or unmanaged shim layer required to pull this off.

The "obvious" solution is to use STDIN and STDOUT and launch a child process - however I don't want a separate process. Also, my attempts to redirect the STDIN and STDOUT streams of a native library which isn't a console application on Windows have failed somewhat spectacularly (and with much head-banging).

Based on this question: Redirect stdout+stderr on a C# Windows service I attempted to modify the approach to (at least) solve the "response" stream half of my problem - but without a FileStream (since I want something more analogous to a MemoryStream). However, FileStream is the only stream type which exposes a suitable low-level stream handle.

Otherwise, I'm pretty well stuck and am currently thinking I'll need to dive deeper and come up with my own hand-rolled native<->managed stream implementation but don't really know where to start.

Solution

Finally!

I've posted a complete sample project here:

http://pastebin.com/jcjHdnwz

This is for .NET 3.5 and uses the AnonymousPipeServerStream - but with a little bit of reflector-ing, it's easy enough to duplicate the inner workings of the AnonymousPipeServerStream in .NET 2.0.

Thanks for your help shf301 for pointing me to the native pipe API, which got me looking into the Microsoft docs for a better understanding of what's going on, and for pointing out I needed to use the _open_osfhandle method to get the FILE * reference.

Community
  • 1
  • 1
Steve
  • 31,144
  • 19
  • 99
  • 122
  • Just replace the `FileStream` with your `MemoryStream`? What matters here is that the stream is wrapped in a `StreamWriter` and that writer is passed in as the new standard out and error streams. – Jeff Mercado Dec 18 '11 at 21:59
  • 1
    Yes, a pipe is required to get a writable handle. .NET 3.5 – Hans Passant Dec 18 '11 at 22:04
  • @JeffMercado, The Win32 function `SetStdHandle` AFAIK requires a handle to a suitable pipe - and `MemoryStream` doesn't provide this - where as `FileStream` does via `.SafeFileHandle.DangerousGetHandle()` – Steve Dec 18 '11 at 22:07
  • @Steve: Ah ok, I didn't know the difference and haven't tried that myself. – Jeff Mercado Dec 18 '11 at 22:18
  • is `FILE *` on the native side an absolute requirement ? – Yahia Dec 21 '11 at 17:47
  • @Steve - see the answer to the following questions go see how to go from a `HANDLE` to a `FILE*`. http://stackoverflow.com/questions/7369445/is-there-a-windows-equivalent-to-fdopen-for-handles – shf301 Dec 21 '11 at 19:21
  • @Yahia, Yes. At the end of the day it's pretty much the singular requirement. – Steve Dec 21 '11 at 19:35
  • @Steve ok... is the maximum siz of the stream known? OR can data that has been read be discarded do the data chunks exchanged have a known maximum size ? – Yahia Dec 21 '11 at 20:10
  • @shf301, I think you're on to something there with the `_open_osfhandle` method - but, I did that using the `ClientSafePipeHandle.DangerousGetHandle()` value and it returns -1. Using the `SafePipeHandle.DangerousGetHandle()` works (in terms of eventually returning a `FILE *`, but then i'm looking at the wrong end of the pipe and nothing works. – Steve Dec 21 '11 at 20:53
  • @yahia, No, the size is unknown until all of the content is read. Yes, the data which is read and processed is then discarded... but I think that detail is orthogonal to the problem at hand. Data chunks can be streamed over the network so I want to grab them and push them out as fast as possible. – Steve Dec 21 '11 at 20:55
  • @Steve I don't understand what the network has to do with it - you write that this about communication between .NET and native within the same process... can you clarify ? – Yahia Dec 21 '11 at 20:58
  • @Yahia, You asked "do the data chunks exchanged have a known maximum size", my answer is: no, not really, because "Data chunks can be streamed over the network so I want to grab them and push them out as fast as possible." which is why I want to interop with the native function with FIFO streams in both directions... on the .NET side of things I'll get getting `Stream` in and writing out to a `Stream`. Also, because it is a cross-platform library, FILE * already works well on other platforms. – Steve Dec 21 '11 at 21:07
  • @Steve understood... for Windows I would have suggested an architecture which outperforms almost anything else... BUT since you say FILE* and chunks of unknown size and cross-platform my idea won't apply... so I can only wish you success in finding what you are looking for... – Yahia Dec 21 '11 at 21:19

1 Answers1

12

You should be able to do this using a AnonymousPipeStream in .NET 3.5 or higher. That exposes a handle, via the SafePipeHandle property that you can pass to SetStdHandle.

For .NET 2.0 you may have to P/Invoke to the unmanaged pipe API.

shf301
  • 31,086
  • 2
  • 52
  • 86
  • I've added a 250 point bounty. Could you expand upon your reference to the "unmanaged pipe API"? – Steve Dec 21 '11 at 03:07
  • @Steve - What I meant is to use P/Invoke to call the unmanaged `CreatePipe` function to create a pipe and `ReadFile` to read from the pipe. See Msdn and pinvoke.net. http://msdn.microsoft.com/en-us/library/windows/desktop/aa365141%28v=vs.85%29.aspx -- http://www.pinvoke.net/default.aspx/kernel32.createpipe – shf301 Dec 21 '11 at 11:21
  • @Steve there is also an named pipe wrapper class for C#. That should probably work as is. http://www.codeguru.com/Csharp/.NET/net_general/performance/article.php/c7259/ – shf301 Dec 21 '11 at 12:38
  • Please see my edit which illustrates what I've tried for this and where I'm failing. – Steve Dec 21 '11 at 17:19
  • You led me to the solution, and I have it working now... give me a bit to package it up and post it. I'll give you credit. – Steve Dec 21 '11 at 21:59