2

I'm trying to create a method in C# to return a string of a web pages html content from the url. I have tried several different ways, but I am getting the error System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.

The following works fine locally, but gets the above error when running on a remote server:

  public static string WebPageRead(string url)
    {
        string result = String.Empty;

        WebResponse response = null;
        StreamReader reader = null;

        try
        {
            if (!String.IsNullOrEmpty(url))
            {
                HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
                request.Method = "GET";
                request.KeepAlive = false;
                request.ProtocolVersion = HttpVersion.Version10;

                response = request.GetResponse();
                reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
                result = reader.ReadToEnd();
            }
        }
        catch (Exception exc)
        {
            throw exc;
        }
        finally
        {
            if (reader != null)
            {
                reader.Close();
            }

            if (response != null)
            {
                response.Close();
            }
        }

        return result;
    }
  • Just to be sure, the problematic URL can be requested using a web browser or another HTTP client? Is there by any chance a proxy set up for Internet Explorer or does the remote server require some authentication such as NTLM? – Lucero Apr 15 '09 at 10:27

3 Answers3

2

This is probably not the problem, but try the following:

public static string WebPageRead(string url)
{
    if (String.IsNullOrEmpty(url))
    {
        return null;
    }

    HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
    if (request == null)
    {
        return null;
    }

    request.Method = "GET";
    request.KeepAlive = false;
    request.ProtocolVersion = HttpVersion.Version10;

    using (WebResponse response = request.GetResponse())
    {
        using (Stream stream = response.GetResponseStream())
        {
            using (StreamReader reader = 
                       new StreamReader(stream, Encoding.UTF8))
            {
                return reader.ReadToEnd();
            }
        }
    }
}

I echo the earlier answer that suggests you try this with a known good URL. I'll add that you should try this with a known good HTTP 1.1 URL, commenting out the line that sets the version to 1.0. If that works, then it narrows things down considerably.

John Saunders
  • 160,644
  • 26
  • 247
  • 397
1

Thanks for the responses, the problem was due to a DNS issue on the remote server! Just to confirm, I went with the following code in the end:

    public static string WebPageRead(string url)
    {
        string content = String.Empty;

        if (!String.IsNullOrEmpty(url))
        {
            HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;

            if (request != null)
            {
                request.Method = "GET";
                request.KeepAlive = false;
                request.ProtocolVersion = HttpVersion.Version10;

                try
                {
                    using (WebResponse response = request.GetResponse())
                    {
                        using (Stream stream = response.GetResponseStream())
                        {
                            using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
                            {
                                content = reader.ReadToEnd();
                            }
                        }
                    }
                }
                catch (Exception exc)
                {
                    throw exc;
                }
            }
        }                    

        return content;
    }
  • 2
    I'm curious what the DNS issue was? I'm getting this same error on our dev machine but not on our prod machine despite identical code and config files. – Mark B Apr 19 '13 at 17:43
  • How do you have fixed the mentioned issue because I still receive that error from your code! – Hamid Dec 06 '13 at 15:59
0

Had a problem like this before that was solved by opening the url in IE on the machine with the problem. IE then asks you whether you want to add the url to the list of secure sites. Add it and it works for that url.

This is just one of the possible causes. Seriously a lot of other problems could cause this. Besides the problem described above, the best way I've found to solve this is the just catch the exception and retry the request.

Fung
  • 7,530
  • 7
  • 53
  • 68