WebClient Timing out when Scheduled in Task

Question

Hey guys this is my first attempt at using the Task libraries in 4.0 so if you see anything else I'm besides my problem that is not correct please do let me know.

My problem is that when I schedule a bunch of task which inside use a webclient to make a request, the first few make it through just fine, but after a certain time my webclient starts throwing an exception. It as if it creates the webclient then sticks in the Task and waits for a thread to pick it up but by that time the timeout time is reached .. that's just my assumption.

Here is the code :

var TmsThread = Task.Factory.StartNew(() => UpdateTmsNullPackages(), TaskCreationOptions.LongRunning);

that runs in the Form1_Load of the windows app. This is what it calls

public void UpdateTmsNullPackages()
    {
        Parallel.ForEach(TmsNullPackages, Package =>
        {
            try
            {
                Task<string> task = Task.Factory.StartNew(() => Package.GetPackageTmsId(), TaskCreationOptions.AttachedToParent);
                task.ContinueWith(t =>
                    {
                        if (!String.IsNullOrEmpty(t.Result))
                        {
                            Package.TMSID = t.Result;
                            NowTmsIdFoundPackages.Add(Package);
                        }
                    });
            }
            catch(Exception ex){}
        });
    }

which in turn, runs this

 public static string GetPackageTmsId(this TwcPackage Package)
    {
        string TMSID = null;
        if (!(String.IsNullOrEmpty(Package.movie_Provider)) && !(String.IsNullOrEmpty(Package.movie_Product)) && !(String.IsNullOrEmpty(Package.movie_Provider_ID)) && !(String.IsNullOrEmpty(Package.movie_Asset_ID)))
        {
            try
            {
                using (WebClient client = new WebClient())
                {
                    client.Credentials = new NetworkCredential(TMSID_Recheck.Properties.Settings.Default.WebRequestUser, TMSID_Recheck.Properties.Settings.Default.WebRequestProdUserPassWord);
                    XmlDocument xmlDoc = new XmlDocument();
                    string URLToBeRequested = TMSID_Recheck.Properties.Settings.Default.RequestProdBaseURL + TMSID_Recheck.Properties.Settings.Default.RequestAPIVersion + "/" + TMSID_Recheck.Properties.Settings.Default.RequestAPIProgramServiceCall + TMSID_Recheck.Properties.Settings.Default.RequestAPIProgramAssociationServiceCall + Package.movie_Provider + ':' + Package.movie_Product + ':' + Package.movie_Provider_ID + "::" + Package.movie_Asset_ID;
                    try
                    {
                        xmlDoc.LoadXml(client.DownloadString(URLToBeRequested));
                        XmlNodeList Program = xmlDoc.DocumentElement.SelectNodes("program");
                        if (Program.Count > 0) TMSID = Program[0].Attributes["TMSId"].Value.ToString();
                    }
                    catch (WebException ex)
                    {
                        if (ex.Status != WebExceptionStatus.Timeout)
                        {
                            if (((HttpWebResponse)ex.Response).StatusCode != HttpStatusCode.NotFound) {  }
                        }
                        else {  }
                    }
                }
            }
            catch (Exception ix) {  }
        }
        return TMSID;
    }

the issue happens when downloadstring is called after a couple hundred tasks it throws a timeout exception.

score 1 · Accepted Answer · edited May 23 '17 at 12:12

1

the issue happens when downloadstring is called after a couple hundred tasks

And how many tasks have completed in that time slice?

It looks like you are simply queuing too many requests. Your system and the remote server probably have policies in place to limit the number of simultaneous connections.

The solution (and a quick diagnosis test) would be to use MaxDegreeOfParallelism in the ForEach.

Here is a similar question with some good answers.

edited May 23 '17 at 12:12

Community

1
1

answered Oct 10 '12 at 17:13

H H

263,252
30
330
514

About 600 get through before the timeouts. Does each task use its own webclient object which is then disposed or do they all use the same ? could you show an example of MaxDegreeOfParallelism . Should i set that to my max processor count ? – metaphysics Oct 10 '12 at 17:17
They each have their own. But the issue is probably threads, not webclients. See the link. – H H Oct 10 '12 at 17:19
1

Parallel.ForEach(TmsNullPackages, new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount * 20 }, Package => im going to try this and see if get timeouts. This will take a bit – metaphysics Oct 10 '12 at 17:29
Good. But note that ProcessorCount is not as important as bandwidth and Connection limits. – H H Oct 10 '12 at 17:31
tried 'new ParallelOptions { MaxDegreeOfParallelism = Environment.ProcessorCount }' that would be 8 but i'm still getting timeouts. – metaphysics Oct 10 '12 at 18:20
You'll have to find out about the policies at both sides. Or setup a TestServer with known properties. But also try with MaxDegree=4. – H H Oct 10 '12 at 18:30
I found out the setting for timeout on the server I'm making the call to using the webclient.. its 600 seconds. The only way this could be hit is ether A. all task use the same webclient or B. all task are schedule at once then the foreach handles one at a time .. this would cause the webrequest to time out in the task at the bottom of the list – metaphysics Oct 10 '12 at 21:38

WebClient Timing out when Scheduled in Task

1 Answers1