Parallel.ForEachAsync help needed

Question

Ok. So I am going to start by apologizing up front as I know there have been a lot of question asked about Parallel and Async. However, even after searching I can not wrap my brain around how this should work. Coding is something I dabble in not something I do day in and day out.

I am trying to help a friend collect some historic stock data using the AlphaVantage API.

I have no issues collecting the data and saving it to the database but it can take a long time to pull 20 years of daily prices. So to avoid the GUI freezing up I made it an async function. However, Considering how often and how much data needs to be pulled it will need to be done in parallel. I am not sure how many parallel events yet but the long term goal is to have a timer in the application that fires off at intervals and calls the BatchUpdateStockHistoryAsync method and passes in a value for how many stocks to pull and update in this batch the function will then go out, query the DB and get a list 5 that have the oldest update.

For the moment I stripped all of that out to simply the question and just created a manual list of 5 stocks that get iterated through. For each stock in the list it calls another function PullAndWriteHistoricDailyData(item) that does all the work of actually reaching out to AlphaVantage and updating the DB.

After this lengthy info my question is what's the best way is to trigger multiple concurrent threads of PullAndWriteHistoricDailyData()?

I imagine I will need to also play with MaxDegreeOfParallelism to figure out what works best but I have not even gotten that far yet. I believe I would want to use Parallel.ForEachAsync maybe? but just not sure how to pull it all together.

Any help would be greatly appreciated

Misiu

    public async Task BatchUpdateStockHistoryAsync()
    {
        List<string> list = new List<string>();

        list.Add("AAPL");
        list.Add("F");
        list.Add("MSFT");
        list.Add("COKE");
        list.Add("IAG");

        foreach(string item in list)
        {
            await Task.Run(() => PullAndWriteHistoricDailyData(item));
        }

        return completed;
    }

For an example of using the `Parallel.ForEachAsync` method, you could look [here](https://stackoverflow.com/questions/15136542/parallel-foreach-with-asynchronous-lambda/68901782#68901782 "Parallel foreach with asynchronous lambda"). — Theodor Zoulias, Apr 08 '22 at 17:22
@ThomasWeller Not at all and I spent a few hours reading up on various things before I posted. But I clearly have a learning deficiency when it comes to this because I was not able to connect the dots. — Misiu02, Apr 08 '22 at 17:33
Since you said `PullAndWriteHistoricDailyData` is calling an external web API (AlphaVantage), you should rewrite *that* method to be async. For example, it could use `HttpClient.GetAsync` to make the call to AlphaVantage. Just wrapping a synchronous method in a `Task.Run` isn't going to magically make it asynchronous. — Matt Johnson-Pint, Apr 08 '22 at 17:44
@MattJohnson-Pint Thank you Matt. I will give that a shot as well. Its actually using GetStringFromURL() but there is a GetStringFromURLAsync() I will try next. — Misiu02, Apr 08 '22 at 18:07

Ryan Wilson · Accepted Answer · 2022-04-08T18:22:04.217

1

EDIT I mistakenly read about Task.WhenAll vs Parallel.ForEach originally. After reading more about Parallel.ForEachAsync based on the answer by Theodore Zoulias at: is-parallel-foreachasync-a-replacement-to-a-plain-for-loop-append-to-task-list, and based on the assumption that you want to view the result of each Task as apposed to receiving a single Task from the call to Parallel.ForEachAsync, then I would propose changing to using the following:

You could switch to using a List<Task> and use await Task.WhenAll() and skip the Parallel.ForEachAsync like so:

 List<Task> myupdatetasks = new List<Task>();

 foreach(string item in list)
 {
     myupdatetasks.Add(Task.Run(() => PullAndWriteHistoricDailyData(item)));
 }

 await Task.WhenAll(myupdatetasks).ConfigureAwait(false);

if PullAndWriteHistoricDailyData() is already an async method, you don't need to use Task.Run, you could do the same as above minus the Task.Run

 List<Task> myupdatetasks = new List<Task>();

 foreach(string item in list)
 {
     myupdatetasks.Add(PullAndWriteHistoricDailyData(item));
 }

 await Task.WhenAll(myupdatetasks).ConfigureAwait(false);

if you are intent on using Parallel.ForEachAsync, this article explains it pretty well: parallelforeachasync-in-net-6

edited Apr 08 '22 at 18:22

answered Apr 08 '22 at 17:06

Ryan Wilson

10,223
2
21
40

Could you name the reasons that the `Task.WhenAll` is preferable to `Parallel.ForEachAsync`? – Theodor Zoulias Apr 08 '22 at 17:24
Thank you Ryan I appreciate it. PullAndWriteHistoricDailyData() is not currently an async method. I will give this a shot now and then mark solved. No idea why I had such a hard time coming to that conclusion on my own. – Misiu02 Apr 08 '22 at 17:35
@TheodorZoulias Although, I'm going to assume you may refer me to your answer here: [/is-parallel-foreachasync-a-replacement-to-a-plain-for-loop-append-to-task-list](https://stackoverflow.com/questions/68544324/is-parallel-foreachasync-a-replacement-to-a-plain-for-loop-append-to-task-list) :) After reading your answer, I would wager that using this as apposed to `Parallel.ForEachAsync` would be desirable if you wanted to inspect the result of each Task instead of getting back a "naked" Task with `Parallel.ForEachAsync` – Ryan Wilson Apr 08 '22 at 18:03
Ryan the OP has explicitly stated that they consider using the `Parallel.ForEachAsync` API, both in the title and the body of the question. So I assumed that you might have some good reasons for suggesting them to skip it, and switch to the `Task.WhenAll` API instead. If the reasons are just the lack of familiarity with .NET 6, you should probably edit the answer clarify this point. – Theodor Zoulias Apr 08 '22 at 18:11
1

@TheodorZoulias I believe I have familiarized myself enough with `Parallel.ForEachAsync` based on your answer I linked to above, and I gave you a reason why this would be a desirable solution based on the hypothetical that the developer wants to see the result of each Task as apposed to getting back what you referred to as a "naked" task via the call to `Parallel.ForEachAsync`, if that isn't a good enough reason, then I guess we can update the answer to add detail that this is what you could do using .Net 5 and below. Or I'll update my answer to include this as a reason for using this. – Ryan Wilson Apr 08 '22 at 18:14

Parallel.ForEachAsync help needed

1 Answers1