What is the fastest and most efficient ways to make a large number of C# WebRequests?

Question

I have a list of URLs (thousands), I want to asynchronously get page data from each URL as fast as possible without putting extreme load on the CPU.

I have tried using threading but it still feels quite slow:

public static ConcurrentQueue<string> List = new ConcurrentQueue<string>(); //URL List (assume I added them already)

public static void Threading()
{
    for(int i=0;i<100;i++) //100 threads
    {
        Thread thread = new Thread(new ThreadStart(Task));
        thread.Start();
    }
}

public static void Task()
{
    while(!(List.isEmpty))
    {
        List.TryDequeue(out string URL);
        //GET REQUEST HERE
    }
}

Is there any better way to do this? I want to do this asynchronously but I can't figure out how to do it, and I don't want to sacrifice speed or CPU efficiency to do so.

Thanks :)

Did you read these; https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/task-parallel-library-tpl https://stackoverflow.com/questions/23737809/running-multiple-tasks-in-parallel — Stefan Wuebbe, Oct 02 '22 at 17:38
Every thread you create has it's own 1MB stack. You're allocating at least 100MB just to get started. — Enigmativity, Oct 03 '22 at 06:42
This rather depends on what you mean by large. However, at any scale I would consider large, the key would be to distribute the requesting across many machines, with many network interfaces, not across threads on one machine. — Jodrell, Oct 03 '22 at 07:01
You can find some solutions in this question: [How to limit the amount of concurrent async I/O operations?](https://stackoverflow.com/questions/10806951/how-to-limit-the-amount-of-concurrent-async-i-o-operations) — Theodor Zoulias, Oct 03 '22 at 09:24

Charlieface · Answer 1 · 2022-10-03T08:37:05.947

You are best off using HttpClient which allows async Task requests.

Just store each task in a list, and await the whole list. To prevent too many requests at once, wait for any single one to complete if there are too many, and remove the completed one from the list.

const int maxDegreeOfParallelism = 100;

static HttpClient _client = new HttpClient();

public static async Task GetAllUrls(List<string> urls)
{
    var tasks = new List<Task>(urls.Count);

    foreach (var url in urls)
    {
        if (tasks.Count == maxDegreeOfParallelism)    // this prevents too many requests at once
            tasks.Remove(await Task.WhenAny(tasks));

        tasks.Add(GetUrl(url));
    }
    await Task.WhenAll(tasks);
}

private static async Task GetUrl(string url)
{
    using var response = await _client.GetAsync(url);
    // handle response here
    var responseStr = await response.Content.ReadAsStringAsync(); // whatever
    // do stuff etc
}

A few syntax errors. This code doesn't look tested. – Enigmativity Oct 03 '22 at 06:44 — Enigmativity, Oct 03 '22 at 06:44

score -1 · Answer 2 · answered Oct 03 '22 at 06:56

You should use Microsoft's Reactive Framework (aka Rx) - NuGet System.Reactive and add using System.Reactive.Linq; - then you can do this:

public static IObservable<(string url, string content)> GetAllUrls(List<string> urls) =>
    Observable
        .Using(
            () => new HttpClient(),
            hc =>
                from url in urls.ToObservable()
                from response in Observable.FromAsync(() => hc.GetAsync(url))
                from content in Observable.FromAsync(() => response.Content.ReadAsStringAsync())
                select (url, content));

That allows you to consume the results in a couple of ways.

You can process them as they get produced:

IDisposable subscription =
    GetAllUrls(urlsx).Subscribe(x => Console.WriteLine(x.content));

Or you can get all of them produced and then await the full results:

(string url, string content)[] results = await GetAllUrls(urlsx).ToArray();

What is the fastest and most efficient ways to make a large number of C# WebRequests?

2 Answers2