Copy http response stream to memory stream without copying all data to memory

Question

I have stream, that comes from a ReadAsStreamSync() method from an HttpContent object. I'm returning this stream from a method, so the caller of my method can read directly from the http stream, without having to buffer the entire response in memory.

The issue I now have, is that I need to know when the stream I return is disposed. Since there is no "disposed" event on a stream, I'm thinking I can wrap the stream in a class that inherits memorystream, and override the dispose method. However, I don't want to copy the ENTIRE response stream to the memorystream, because this would mean the entire content would be written to memory. I think this is possible, but I can't seem to find any info about it.

Is there anyone who can help me with this? Or provide me with an alternative method to be able to tell when a stream is being disposed.

Edit:

After reading the comments it seems that I haven't been very clear. Let me give you a concrete example:

public class StreamGetter()
{
    public Stream GetSomeStream()
    {
        //some code here that gets the stream

        //and return the stream, this stream might be 100mb long, 
        //and since this method doesn't know what to do with it, 
        //we return the stream so the calling method can figure it out
        return generatedStream;
    }
}

public void SomeOtherMethod()
{
    using(var stream = GetSomeStream())
    {
        //some some stuff
    }
}

Now the thing is, the stream that is fetched is a stream that comes from a ReadAsStreamAsync() of the content of an httpResponseMessage, which is fetched using an HttpClient. Now I need to dispose the HttpClient once the stream is disposed. If I dispose it earlier, I can no longer read from the stream. If I don't dispose it, the resources are never freed.

Now before anyone starts mentioning i shouldn't dispose HttpClient because it is made for re-usage. I know about it, and I don't reuse HttpClient, but the underlying HttpClientHandler.

If you think my design is a bad design, I'm also open to suggestions on how to write the code differently. The only requirement is that I keep all code related to the HttpClient in the "StreamGetter" class, and all handling of the result stream outside of the "StreamGetter" class.

edit2: I think I found a solution by changing the class. By making the StreamGetter class implement idisposable, I can move the responsibility to dispose stuff outside of the StreamGetter class. The calling class can now create a using block, call the "GetSomeStream" method, and process the results all within the using block. The dispose method of the StreamGetter then disposes the HttpClient. Any thoughts on this?

Why do you need to know if the stream is disposed? If you are consuming the stream synchronously, shouldn't the next call outside of the using block (if you are using one) indicate the stream has been disposed? — Phil Cooper, Nov 02 '18 at 12:55
I'm not consuming the stream in my method. Another method can call my method, and my method returns the stream. The calling method then processes the stream later. — PaulVrugt, Nov 02 '18 at 12:59
You need to show examples its hard to know what you are doing — TheGeneral, Nov 02 '18 at 13:04
The whole point of a memorystream *is* to hold its entire contents in memory. It's extremely unclear why you need to interpose another stream here at all. What is the nature of the actions you need to take when this "stream" is disposed? — Damien_The_Unbeliever, Nov 02 '18 at 13:09
@PaulVrugt "I'm not consuming the stream in my method. Another method can call my method, and my method returns the stream. The calling method then processes the stream later." That sounds like a terrible design mistake. The more you talk, the more this sounds like a XY Problem. Please go a few steps back and explain us what problem you even try to solve with that wierd design. — Christopher, Nov 02 '18 at 13:10
HTTP uses TCP as the transport layer. There are two solutions using TCP 1) Use the Keep-Alive event. When a connection closes The keep alive will indicate when connection closes. 2) Capture the Close Event of the TCP connection. — jdweng, Nov 02 '18 at 13:22
Rather than subclassing `MemoryStream`, you could follow the [decorator pattern](https://en.wikipedia.org/wiki/Decorator_pattern) and create some `StreamDecorator` that subclasses `Stream` and wraps an underlying stream. Then, implement some `OnClosed` event in the decorator. — dbc, Nov 02 '18 at 16:27
For some existing `Stream` decorators see [`StreamProxy`](http://www.dotnetframework.org/default.aspx/DotNET/DotNET/8@0/untmp/WIN_WINDOWS/lh_tools_devdiv_wpf/Windows/wcp/TrustUi/MS/Internal/documents/Application/StreamProxy@cs/1/StreamProxy@cs), [`BufferedStream`](https://referencesource.microsoft.com/#mscorlib/system/io/bufferedstream.cs), [FileStream disable Close()](https://stackoverflow.com/q/20097577), and [Stream wrapper to make Stream seekable?](https://stackoverflow.com/a/28036366). — dbc, Nov 02 '18 at 16:30
@dbc nice suggestion. However I think my refactoring the code to be disposable is a solution better fitting this particular issue. You suggestion might come in handy for future cases though! — PaulVrugt, Nov 02 '18 at 20:37
@Christopher I don't see how returning a stream from a function is a bad idea in general. A bit more constructive and less condescending comment would be appreciated. However, my own suggestion making the class itself disposable seems the most elegant solution so far, but you might have another? — PaulVrugt, Nov 02 '18 at 20:42
@PaulVrugt: My simple rule is to never split up the creation and disposing of any Disposeable Resource. Create. Use. dispose. All in the same peice of code, ideally using a using. Failing that? Wrap it into a Disposeable class whose sole purpose is to relay the Dispose call. At wich point the question is, why you are not just using the stream itself to begin with? — Christopher, Nov 03 '18 at 15:21
Yeah, this discussion in going into the wrong direction. People should just answer the question being asked instead of asking "why would you want to this". @Christopher : I'm not using the stream because the method calling the http request has no idea what the stream contains or what to do with it,because it is a reusable method to make a http request. I'm going to go with making the class itself disposable — PaulVrugt, Nov 04 '18 at 12:29
@PaulVrugt: If you need to encapsulate a Stream in another, just use the proper Constructor. This operation is trivial. With anything else, we have issues even figuring out what your problem is or how you got into that potentially very dangerous solution. — Christopher, Nov 04 '18 at 17:21

score 0 · Accepted Answer · answered Nov 04 '18 at 12:31

0

In the end I went with making the entire class disposable, and in doing this, moving the responsibility of disposing stuff outside of the class and into the calling method. This removed the necessity of having to know when the stream is disposed

answered Nov 04 '18 at 12:31

PaulVrugt

1,682
2
17
40

You did relay the dispose call to teh contained Stream? Just asking, because that is the main reason we chain Disposeable classes like that. – Christopher Nov 04 '18 at 17:21
the dispose of the StreamGetter class disposes the underlying httpclient. The httpclient will in turn dispose of the response stream – PaulVrugt Nov 05 '18 at 09:50

Copy http response stream to memory stream without copying all data to memory

1 Answers1