C# .NET Parallel I/O operation (with throttling)

Question

I'm trying to figure out the best approach to perform I/O in parallel. For quite long time I even thought that I understand this topic and there's nothing what could surprise me. But I was wrong.

Question: What is the best way to perform I/O in parallel (e.g. downloading feed via HTTP), taking in consideration that service may start refusing requests if there are too many of them?

Before I write any code...

• when work I want to perform is some sort of computation, I want to use CPU (CPU bound task) -> PLINQ is the best (easiest) option
• when work I want to perform is some sort of I/O operation, I want to use asynchronous model (await/async) to not block main thread (allowing web server handle more requests while some requests are just waiting for I/O)
• optimal performance is when number of CPU equals to number of threads
• when performing I/O without async model, I'm basically blocking thread until it gets response
• threads are expensive (memory allocation, thread management, context switching)
• Task is abstraction over thread: multiple Tasks may or may not be performed with one thread
• TaskScheduler takes care of queuing work to threads, thread pool manages number of threads according to current environment and application needs
• there still must be place where I await my Task to not block executing thread (so I can reuse thread for other task while waiting for response)

Solution?

I came with various ways how to consume feed, but basically almost all of them were not throttled, possibly causing overload of service.

1. awaiting sequentially in foreach
2. awaiting Task.WhenAll(taks)
3. preparing tasks in parallel and then awaiting them all again
4. parallel blocking .AsParallel().Select(t => t.Result)
5. using asynchronous .ForEachAsync described by Stephen Toub: Implementing a simple ForEachAsync, part 2, storing results in ConcurrenyBag

(5) Queue and batches using tasks (throttling)

Allowing just few tasks to run. Similarly to #4, this requires some fiddling to find optimal performance.

    {
        var result = new List<DummyDelay>(Total);
        var queue = new List<Task<DummyDelay>>(Parallelism);
        for (var i = 0;; i++)
        {
            // 1. enqueue work
            if (i < Total)
            {
                queue.Add(LoadDummyAsync(Delay, Total, i));
            }

            // 2. no more work, break
            if (queue.Count == 0)
            {
                break;
            }

            // 3. if queue is big enough, await one result
            if (queue.Count == Parallelism || i > Total)
            {
                Task<DummyDelay> finishedTask = await Task.WhenAny(queue);
                queue.Remove(finishedTask);

                result.Add(finishedTask.Result);
            }
        }

        return result;
    }

(6) Producer-consumer pattern (throttling ?)

(not implemented) I'm bit skeptic because orchestration around producer-consumer will cost something, and I will still have to await somewhere.