Is there a built-in way to handle multiple files as one stream?

Question

I have a list of files, and i need to read them each in a specific order into byte[] of a given size. This in itself is not a problem for a single file, a simple while ((got = fs.Read(piece, 0, pieceLength)) > 0) gets the job done perfectly fine. The last piece of the file may be smaller than desired, which is fine.

Now, there is a tricky bit: If I have multiple files, I need to have one continous stream, which means that if the last piece of a file is smaller that pieceLength, then I need to read (pieceLength-got) of the next file, and then keep on going on until the end of the last file.

So essentially, given X files, I will always read pieces that are exactly pieceLength long, except for the very last piece of the very last file, which may be smaller.

I just wonder if there is already something build in .net (3.5 SP1) that does the trick. My current approach is to create a Class that takes a list of files and then exposes a Read(byte[] buffer, long index, long length) function, similar to FileStream.Read(). This should be pretty straight forward because I do not have to change my calling code that reads the data, but before I reinvent the wheel I'd just like to double check that the wheel is not already built into the BCL.

Thanks :)

Answer 1

I don't believe there's anything in the framework, but I'd suggest making it a bit more flexible - take an IEnumerable<Stream> in your constructor, and derive from Stream yourself. Then to get file streams you can (assuming C# 3.0) just do:

Stream combined = new CombinationStream(files.Select(file => File.Open(file));

The "ownership" part is slightly tricky here - the above would allow the combination stream to take ownership of any stream it reads from, but you may not want it to have to iterate through all the rest of the streams and close them all if it's closed prematurely.

Answer 2

Here is what I came up based on @jon skeet's idea.

It just implements Read which was quite sufficient for me. (but no i need help on implementing the BeginRead/EndRead method.) Here is the full code containing both sync and async - Read and BeginRead/EndRead

https://github.com/facebook-csharp-sdk/combination-stream/blob/master/src/CombinationStream-Net20/CombinationStream.cs

internal class CombinationStream : System.IO.Stream
{
    private readonly System.Collections.Generic.IList<System.IO.Stream> _streams;
    private int _currentStreamIndex;
    private System.IO.Stream _currentStream;
    private long _length = -1;
    private long _postion;

    public CombinationStream(System.Collections.Generic.IList<System.IO.Stream> streams)
    {
        if (streams == null)
        {
            throw new System.ArgumentNullException("streams");
        }

        _streams = streams;
        if (streams.Count > 0)
        {
            _currentStream = streams[_currentStreamIndex++];
        }
    }

    public override void Flush()
    {
        if (_currentStream != null)
        {
            _currentStream.Flush();
        }
    }

    public override long Seek(long offset, System.IO.SeekOrigin origin)
    {
        throw new System.InvalidOperationException("Stream is not seekable.");
    }

    public override void SetLength(long value)
    {
        this._length = value;
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        int result = 0;
        int buffPostion = offset;

        while (count > 0)
        {
            int bytesRead = _currentStream.Read(buffer, buffPostion, count);
            result += bytesRead;
            buffPostion += bytesRead;
            _postion += bytesRead;

            if (bytesRead <= count)
            {
                count -= bytesRead;
            }

            if (count > 0)
            {
                if (_currentStreamIndex >= _streams.Count)
                {
                    break;
                }

                _currentStream = _streams[_currentStreamIndex++];
            }
        }

        return result;
    }

    public override long Length
    {
        get
        {
            if (_length == -1)
            {
                _length = 0;
                foreach (var stream in _streams)
                {
                    _length += stream.Length;
                }
            }

            return _length;
        }
    }

    public override long Position
    {
        get { return this._postion; }
        set { throw new System.NotImplementedException(); }
    }
    public override void Write(byte[] buffer, int offset, int count)
    {
        throw new System.InvalidOperationException("Stream is not writable");
    }

    public override bool CanRead
    {
        get { return true; }
    }

    public override bool CanSeek
    {
        get { return false; }
    }

    public override bool CanWrite
    {
        get { return false; }
    }
}

Also available as a NuGet package

Install-Package CombinationStream