Parallel http requests

Question

I have an app that making API requests to the last.fm website using the backgroundWorker. Initially I don't know how many request I'm gonna need to make. The response contains the total number of pages, so I will only get it after the first request. Here is the code below.

    private void backgroundWorker_DoWork(object sender, DoWorkEventArgs e)
    {            
        int page = 1;
        int totalpages = 1;

        while (page <= totalpages)
        {
            if (backgroundWorker.CancellationPending)
            {
                e.Cancel = true;
                return;
            }

            //Here is the request part
            string Response = RecentTracksRequest(username, from, page);

            if (Response.Contains("lfm status=\"ok"))
            {
                totalpages = Convert.ToInt32(Regex.Match(Response, @"totalPages=.(\d+)").Groups[1].Value);

                MatchCollection match = Regex.Matches(Response, "<track>((.|\n)*?)</track>");
                foreach (Match m in match)
                    ParseTrack(m.Groups[1].Value);
            }
            else
            {
                MessageBox.Show("Error sending the request.", "Error",
                    MessageBoxButtons.OK, MessageBoxIcon.Error);
                return;
            }

            if (page >= totalpages)
                break;

            if (totalpages == 0)
                break;

            if (page < totalpages)
                page++;
        }

The problem is that the last.fm API is really slow and it might take up to 5 seconds to get the responce. With a big number of pages, the loading will take a long time.

I want to make parallel requests, say 3 parallel requests at a time. Is it possible? If yes, how can I make it?

Thanks a lot.

BTW, if you're making requests to the same host (last.fm in your case) .NET will limit the number of concurrent http requests. See accepted answer here: http://stackoverflow.com/questions/1361771/max-number-of-concurrent-httpwebrequests — Dmitry, Oct 31 '12 at 09:54

cuongle · Accepted Answer · 2017-02-28T10:04:11.797

11

You can take advantage of HttpClient, assume you have list of urls:

var client = new HttpClient();
var tasks = urls.Select(url => client.GetAsync(url).ContinueWith(t =>
            {
                var response = t.Result;
                response.EnsureSuccessStatusCode();

                //Do something more
            }));

If you use async method, you can await all tasks finish like below:

var results = await Task.WhenAll(tasks);

edited Feb 28 '17 at 10:04

answered Oct 31 '12 at 09:51

cuongle

74,024
28
151
206

Another way of `Parallel.ForEach` – L.B Oct 31 '12 at 10:06

score 2 · Answer 2 · answered Oct 31 '12 at 09:48

You can do Async Web Request as well, using BeginGetResponse

      HttpWebRequest webRequest;
      webRequest.BeginGetResponse(new AsyncCallback(callbackfunc), null);


      void callbackfunc(IAsyncResult response)
      {
         webRequest.EndGetResponse(response);
      }

Parallel http requests

2 Answers2

Linked