Working with multiple http request-response

Question

I have a program which loops through an apps list.

Apps
--------
App1
App2
App3

Now, for each of them, I do a http request to get a list of builds for each app as an Xml.

So a request like,

http://example.com/getapplist.do?appid=App1

gives me a response like,

<appid name="App1">
  <buildid BldName="Bld3" Status="Not Ready"></buildid> 
  <buildid BldName="Bld2" Status="Ready"></buildid>
  <buildid BldName="Bld1" Status="Ready"></buildid>
</appid>

Now I get the Highest build number with Status "Ready" and then do another web api call like,

http://example.com/getapplist.do?appid=App1&bldid=Bld2

This gives me a response like,

 <buildinfo appid="App1" buildid="Bld2" value="someinfo"></build>

I feed these into internal data tables. But now, this program takes a painfully long time to complete (3 hours), since I have close to 2000 appids and there are 2 Web requests for each id. I tried sorting this issue using a BackgroundWorker as specified here. I thought of collating all info from http responses into a single XML and then using that XML for further processing. This throws the error,

file being used by another process

So my code looks like,

if (!backgroundWorker1.IsBusy) 
{
    for(int i = 0; i < appList.Count; i++)
    { 
        BackgroundWorker bgw = new BackgroundWorker();
        bgw.WorkerReportsProgress = true;  
        bgw.WorkerSupportsCancellation = true;                     
        bgw.DoWork += new DoWorkEventHandler(bgw_DoWork);                   
        bgw.ProgressChanged += new ProgressChangedEventHandler(bgw_ProgressChanged);
        bgw.RunWorkerCompleted += new RunWorkerCompletedEventHandler(bgw_RunWorkerCompleted);
        //Start The Worker 
        bgw.RunWorkerAsync();
    }
}

And the DoWork function picks the tag values and puts it into an XML.

What is the best way I can get the app- buildinfo details into a common file from all the http responses from all the background workers?

Answer 1

HTTP requests are IO bound and asynchronous by nature, there is no reason to use background workers to accomplish what you need.

You can take advantage of async-await which is compatible in .NET 4 via Microsoft.Bcl.Async and HttpClient:

private async Task ProcessAppsAsync(List<string> appList)
{
    var httpClient = new HttpClient();

    // This will execute your IO requests concurrently,
    // no need for extra threads.
    var appListTasks = appList.Select(app => httpClient.GetAsync(app.Url)).ToList();

    // Wait asynchronously for all of them to finish
    await Task.WhenAll(appListTasks);

   // process each Task.Result and aggregate them to an xml
    using (var streamWriter = new StreamWriter(@"PathToFile")
    {
        foreach (var appList in appListTasks)
        {
           await streamWriter.WriteAsync(appList.Result);
        }
    }
}

This way, you process all requests concurrently and handle results from all of them once they've completed.

Answer 2

This solution works for .Net 2.0 and up by using the async methods from the WebClient class and using a counter that is decremented with the Interlocked class and ordinary lock to serialize the writing of the results to the file.

var writer = XmlWriter.Create(
    new FileStream("api.xml",
                    FileMode.Create));
writer.WriteStartElement("apps"); // root element in the xml
// lock for one write
object writeLock = new object(); 
// this many calls            
int counter = appList.Count;

foreach (var app in appList)
{
    var wc = new WebClient();

    var url = String.Format(
        "http://example.com/getapplist.do?appid={0}&bldid=Bld2", 
        app);
    wc.DownloadDataCompleted += (o, args) =>
        {
            try
            {
                var xd = new XmlDocument();
                xd.LoadXml(Encoding.UTF8.GetString(args.Result));
                lock (writeLock)
                {
                    xd.WriteContentTo(writer);
                }
            }
            finally
            {
                // count down our counter in a thread safe manner
                if (Interlocked.Decrement(ref counter) == 0)
                {
                    // this was the last one, close nicely
                    writer.WriteEndElement();
                    writer.Close();
                    ((IDisposable) writer).Dispose();
                }
            }
        };
    wc.DownloadDataAsync(
        new Uri(url));   
}