C# Run multiple tasks that parse website and return when they all finish

Question

My objective is to load multiple links at the same time and create a task for each of them.

The task will call an async method that will parse the links and returns sublinks, which in return will be parsed (using WebBrowser) and then they return a download link.

The first async method will call 2 subsequent methods for that work to be done.

My problem is Task.Factory.ContinueWhenAll would return only when the all first method finish, and won't wait for the rest of the work to be done. I only want to continue when I have all download links ready which may need multiple webpage parsings before they are.

Currently my code is the following:

var tasks = new List<Task>();
for (var index = 0; index < items_checklist.CheckedItems.Count; index++)
{
    var item = items_checklist.CheckedItems[index];
    Task task = Task.Factory.StartNew(
                    () => GetMirrors(((Item) item).Value, ((Item) item).Text)
                    , CancellationToken.None
                    , TaskCreationOptions.None
                    , TaskScheduler.FromCurrentSynchronizationContext()
    );
    tasks.Add(task);
 }

Task.Factory.ContinueWhenAll(tasks.ToArray(), GetLinks_Finished =>
{
    SetLinksButtonText(@"Links Ready");
    SetLinksButtonState(false);
    SetDownloadButtonState(true);
    Cursor.Current = DefaultCursor;
});

This will return when all GetMirrors finish but GetMirrors would call "tempbrowser_DocumentCompleted" (WebBrowser complete event) which in turn would call "LoadLinkIntoQueue" to load the download link into the queue.

I want ContinueWhenAll to resume when all LoadLinkIntoQueue are executed.

What is my logic missing?

Answer 1

You can create a TaskCompletionSource in your GetMirrors method, which is the method used in the Task.Factory.StartNew call inside your for loop of urls to process.

In GetMirrors you would hook up the DocumentCompleted event of a new WebBrowser which will call the SetResult on the TaskCompletionSource causoing the task to transition to Completed.

Your implementation would be like this:

Task<string> GetMirrors(string url, string somethingelse )
{

    // this will signal that the Task is completed
    // we want the parent to wait
    var tcs = new TaskCompletionSource<string>(TaskCreationOptions.AttachedToParent);

    // give each task their own WebBrowser instance
    WebBrowser tempbrowser = new WebBrowser();
    tempbrowser.ScriptErrorsSuppressed = true;
    this.Controls.Add(tempbrowser);

    tempbrowser.DocumentCompleted += (s, e) => {
        // LoadLinkIntoQueue call 
        // we have a result so signal to the CompletionSource that we're done
        tcs.SetResult(e.Url.ToString());

        this.Controls.Remove(tempbrowser);
    };

    // hook up errorhandling if you need that, left as an exercise.

    tempbrowser.Navigate(url);
    // we return the Task from the completion source
    return tcs.Task ; 
}

You can also call SetException on the TaskCompletionSource instance if you want to return exceptions that occur.

Notice that in this code I instantiate a WebBrowser for each task, so you don't have to worry about serializing the tasks to only have a single WebBrowser control handle a task.