Is it possible to asynchronously enumerate over chunks of file bytes?

Question

What I'm trying to do is like

    /// <summary>
    /// Read zip file into chunks of bytes of largest size possible
    /// </summary>
    /// <param name="stream"></param>
    /// <returns>Enumeration of byte arrays</returns>
    private static async Task<IEnumerable<byte[]>> ChunkStreamBytes(FileStream stream)
    {
        var tasks = new List<Task<byte[]>>();
        for (long bytesRemaining = stream.Length; bytesRemaining > 0;)
        {
            int chunkSize = (int)Math.Min(bytesRemaining, int.MaxValue);
            byte[] chunk = new byte[chunkSize];
            bytesRemaining -= await stream.ReadAsync(chunk, 0, chunk.Length);
            yield return chunk;
        }
    }

but I get the error

The body of ... cannot be an iterator block because Task> is not an iterator interface type

I thought of trying await Task.WaitAll(...) where each task is reading a chunk, but I'm not sure if the tasks can run out of order and screw up the file I'm trying to construct. I need to use async-await pattern because of the context. Any ideas on the proper solution?

[Async Streams](https://github.com/dotnet/csharplang/blob/master/proposals/async-streams.md) is, I believe, currently suggested for possible inclusion in C# 8.0: "C# has support for iterator methods and async methods, but no support for a method that is both an iterator and an async method.." — Damien_The_Unbeliever, Nov 08 '17 at 15:33
This may be supported in C# 8, for now you can try the Reactive extensions to do async streams. — Stefano d'Antonio, Nov 08 '17 at 15:33

score 2 · Answer 1 · answered Nov 08 '17 at 15:33

The IEnumerable<T> / IEnumerator<T> API (including the duck-typed equivalent) is inherently synchronous; you cannot make it async trivially, because it essentially unpacks into a bool MoveNext() and T Current {get;} pair - neither of which is async. There is much discussion over an async enumerable pattern soon, but: not today.

Is it possible to asynchronously enumerate over chunks of file bytes?

1 Answers1