Get webpage page content and HTTP status code in C#

Question

In a C# Windows Forms application I can get the contents of a webpage using:

string content = webClient.DownloadString(url);

And I can get the HTTP header using:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
string response = ((HttpWebResponse)request.GetResponse()).StatusCode.ToString();

Is there a way to get both the contents and the HTTP status code (if it fails) in one trip to the server instead of twice?

Thanks.

`request.GetResponse()` is getting you both. You're the one who takes only `StatusCode` from it. — Tobberoth, Nov 26 '13 at 11:50

Miguel Angelo · Accepted Answer · 2013-11-26T12:16:11.860

You can read the data from the Stream inside the HttpWebResponse object:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
using (var reader = new StreamReader(stream))
{
    HttpStatusCode statusCode = ((HttpWebResponse)response).StatusCode;
    string contents = reader.ReadToEnd();
}

In this way you will have to detect the encoding by hand, or using a library to detect encoding. You can read the encoding as a string from the HttpWebResponse object as well, when one exists, it is inside the ContentType property. If the page is Html, then you will have to parse it for a possible encoding change in the top of the document or inside the head.

Read handling the encoding from ContentType header

var request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
string content;
HttpStatusCode statusCode;
using (var response = request.GetResponse())
using (var stream = response.GetResponseStream())
{
    var contentType = response.ContentType;
    Encoding encoding = null;
    if (contentType != null)
    {
        var match = Regex.Match(contentType, @"(?<=charset\=).*");
        if (match.Success)
            encoding = Encoding.GetEncoding(match.ToString());
    }

    encoding = encoding ?? Encoding.UTF8;

    statusCode = ((HttpWebResponse)response).StatusCode;
    using (var reader = new StreamReader(stream, encoding))
        content = reader.ReadToEnd();
}

`GetResponse and GetResponseStream` can throw ***exceptions*** — PreguntonCojoneroCabrón, Mar 23 '18 at 20:52
Why not ***CharacterSet*** ? `var encoding = response.CharacterSet == "" ? Encoding.UTF8 : Encoding.GetEncoding(response.CharacterSet);` — PreguntonCojoneroCabrón, Mar 23 '18 at 21:13
I don't remember if at the time of this answer `CharacterSet` property was there. — Miguel Angelo, Mar 23 '18 at 22:28

score 6 · Answer 2 · edited May 23 '17 at 12:25

WebClient

I assume you use WebClient because its easy webrequest-to-string handling. Unfortunately, WebClient does not expose the HTTP response code. You can either assume the response was positive (2xx) unless you get an exception and read it:

try
{
    string content = webClient.DownloadString(url);
}
catch (WebException e)
{
    HttpWebResponse response = (System.Net.HttpWebResponse)we.Response;     
    var statusCode = response.StatusCode;
}

Or if you're really interested in the success code you can use reflection as explained here.

HttpClient

You can also use HttpClient if you're on .NET 4.5, which does expose the response code, as explained here:

using (HttpClient client = new HttpClient())
{
    HttpResponseMessage response = await client.GetAsync(url);

    string content = await response.Content.ReadAsStringAsync();
    var statusCode = response.StatusCode;       
}

HttpWebRequest

Alternatively, you can just use HttpWebRequest to get the status and response as explained here:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.Method = "GET";
var response = (HttpWebResponse)request.GetResponse();

using (Stream stream = response.GetResponseStream())
{
   StreamReader reader = new StreamReader(stream);

   string content = reader.ReadToEnd();
   var statusCode = response.StatusCode;    
}

`GetResponse and GetResponseStream` can throw ***exceptions*** — PreguntonCojoneroCabrón, Mar 23 '18 at 20:52
GetResponse and GetResponseStream, --will-- throw exception in case of server error (like 404), and you'll not get the content... — BitQuestions, Apr 15 '19 at 14:11

score 0 · Answer 3 · answered Nov 26 '13 at 11:50

I think, you have not realised, that in the second case you have access to the content as well (although it takes a little more effort to get as a string).

Look at the Microsoft documentation: http://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.getresponsestream(v=vs.110).aspx which shows you how to ge a response stream from the web response, and then how to get the string data from that stream.

score 0 · Answer 4 · answered Nov 26 '13 at 11:53

And I can get the HTTP header using: request.Method = "GET";

Method GET returns HEAD and BODY sections in response. HTTP also support a method HEAD - which returns HEAD section only.

You can get BODY from HttpWebResponse using GetResponseStream method.

Get webpage page content and HTTP status code in C#

4 Answers4