API call getting timed out for large amout of data

Question

I am having trouble in fetching XML data from through API in large amount. Has anyone a better solution in fetching XML data in ASP.net application?

Maybe the issue is might be garbage collection memory not sure but earlier it was not working for any user so I added manual memory release code and closing web response connection after each call.

Any suggestion to improve the fetching of large amount of data are welcome?

   public static XmlDocument LoadXmlFromUrl(string url, bool keepAlive = true, bool throwException = false, int timeout = 15000)
    {
        Log(url, "*** start ***" + url);

        ServicePointManager.DefaultConnectionLimit = 2000;
        string xmlAsText = null;
        HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
        req.KeepAlive = keepAlive;
        req.Timeout = timeout; // 30,000ms = 30 sec default
        try
        {
            bool retried = false;
        retry:
            try
            {
                using (HttpWebResponse webResponse = (HttpWebResponse)req.GetResponse())
                {
                    using (StreamReader responseStream = new StreamReader(webResponse.GetResponseStream(), Encoding.UTF8))
                    {
                        xmlAsText = responseStream.ReadToEnd();
                        responseStream.Close();
                    }
                    webResponse.Close();

                    Log(url, "stream reader (responseStream.ReadToEnd) completed to string.");
                }
            }
            catch (WebException webEx)
            {
                if (!retried)
                {
                    if (webEx.Status == WebExceptionStatus.Timeout || webEx.Status == WebExceptionStatus.KeepAliveFailure)
                    {
                        Log(url, "About to retry for url (6 sec wait): " + url);
                        System.Threading.Thread.Sleep(6000);
                        retried = true;
                        goto retry;
                    }
                }
                else
                {
                    throw webEx;
                }
            }
        }
        catch (Exception ex)
        {
            Log(url, "exception while loading responseStream.ReadToEnd. " + ex.ToString());
        }
        XmlDocument xdoc = null;
        if (!string.IsNullOrEmpty(xmlAsText))
        {
            try
            {
                xdoc = new XmlDocument();
                xdoc.LoadXml(xmlAsText);
                Log(url, "string added to xml (xdoc.LoadXml) completed.");
            }
            catch (Exception ex)
            {
                Log(url, "exception while loading xdoc.LoadXml. " + ex.ToString());
            }
        }
        return xdoc;
    }

Exception 1:

5/01/2017 12:27:01 p.m. exception while loading responseStream.ReadToEnd.     System.Net.WebException: The operation has timed out**

Exception 2:

6/01/2017 8:35:59 a.m. exception while loading responseStream.ReadToEnd. System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

You seem to set up wrong timeout because, if understand problem correctly, your request establishes connection with server but fails during receiving data. You need to increase HttpWebRequest.ReadWriteTimeout. Here is a pretty good explanation http://stackoverflow.com/questions/7250983/httpwebrequests-timeout-and-readwritetimeout-what-do-these-mean-for-the-unde — Yuriy Tseretyan, Dec 21 '16 at 02:41
Thanks @YuriTceretia working in local machine but still need to check in the live environment. I will let you know. Mary Christmas — Mack Patel, Dec 21 '16 at 23:16
Can you please update your question with exception that is thrown during request executing? — Yuriy Tseretyan, Jan 05 '17 at 14:19
6/01/2017 9:52:57 a.m. Stack trace= at System.String.GetStringForStringBuilder(String value, Int32 startIndex, Int32 length, Int32 capacity) at System.Text.StringBuilder.GetNewString(String currentString, Int32 requiredLength) at System.Text.StringBuilder.Append(Char[] value, Int32 startIndex, Int32 charCount) at System.IO.StreamReader.ReadToEnd() at UCResearchWebService.ApiFetcher.LoadXmlFromUrl(String url, Boolean keepAlive, Boolean throwException, Int32 timeout) in c:\Local\TFS2012\Spark Webservice\MAIN\UCResearchWebService\ApiFetcher.cs:line 57 — Mack Patel, Jan 05 '17 at 20:54
Ok, it does not tell anything. Try to trace response using MSDN article I provided. — Yuriy Tseretyan, Jan 05 '17 at 20:57

score 0 · Answer 1 · answered Jan 05 '17 at 20:38

Exception #1 may occur if

request is huge. In this case you can try to increase value of HttpWebRequest.ReadWriteTimeout to something relatively big, for example, 3600000 (1 hour).
value of header ContentLength does not match real response size and server does not force connection to close.

To track down the problem I would suggest you to set up tracing. You can follow https://msdn.microsoft.com/en-us/library/ty48b824(v=vs.110).aspx or find similar here on StackOverflow.

Exception #2 definitely occurs just because the response does not fit to memory allocated for application or its size exceeded 2Gb. (It must be really big response). The possible solution is to buffer stream to a file or process it on-fly (if possible). For example:

...
using (HttpWebResponse webResponse = (HttpWebResponse)req.GetResponse())
{
    String tempFile = Path.GetTempFileName();
    using (var stream = webResponse.GetResponseStream())
    {
        if (stream != null)
        {
            using (FileStream file = File.Create(tempFile))
            {
                stream.CopyTo(file);
            }
        }
    }

    Log(url, "stream reader (responseStream.ReadToEnd) completed to string.");
}

API call getting timed out for large amout of data

1 Answers1