Lazy stream for C# / .NET

Question

Does anyone know of a lazy stream implementation in .net? IOW, I want a to create a method like this:

public Stream MyMethod() {
    return new LazyStream(...whatever parameters..., delegate() {
        ... some callback code.
    });
}

and when my other code calls MyMethod() to return retrieve the stream, it will not actually perform any work until someone actually tries to read from the stream. The usual way would be to make MyMethod take the stream parameter as a parameter, but that won't work in my case (I want to give the returned stream to an MVC FileStreamResult).

To further explain, what I'm looking for is to create a layered series of transformations, so

Database result set =(transformed to)=> byte stream =(chained to)=> GZipStream =(passed to)=> FileStreamResult constructor.

The result set can be huge (GB), so I don't want to cache the result in a MemoryStream, which I can pass to the GZipStream constructor. Rather, I want to fetch from the result set as the GZipStream requests data.

Answer 1

Most stream implementations are, by nature, lazy streams. Typically, any stream will not read information from its source until it is requested by the user of the stream (other than some extra "over-reading" to allow for buffering to occur, which makes stream usage much faster).

It would be fairly easy to make a Stream implementation that did no reading until necessary by overriding Read to open the underlying resource and then read from it when used, if you need a fully lazy stream implementation. Just override Read, CanRead, CanWrite, and CanSeek.

Answer 2

In your Stream class you have to implement several methods of System.IO.Stream including the Read method.

What you do in this method is up to you. If you choose to call a delegate - this is up to you as well, and of course you can pass this delegate as one of the parameters of your constructor. At least this is how I would do it.

Unfortunately it will take more than implementing read method, and your delegate will not cover other required methods

Answer 3

This answer (https://stackoverflow.com/a/22048857/1037948) links to this article about how to write your own stream class.

To quote the answer:

The producer writes data to the stream and the consumer reads. There's a buffer in the middle so that the producer can "write ahead" a little bit. You can define the size of the buffer.

To quote the original source:

You can think of the ProducerConsumerStream as a queue that has a Stream interface. Internally, it's implemented as a circular buffer. Two indexes keep track of the insertion and removal points within the buffer. Bytes are written at the Head index, and removed from the Tail index.

If Head wraps around to Tail, then the buffer is full and the producer has to wait for some bytes to be read before it can continue writing. Similarly, if Tail catches up with Head, the consumer has to wait for bytes to be written before it can proceed.

The article goes on to describe some weird cases when the pointers wrap around, with full code samples.