81

How I can get the content of the web page using ASP.NET? I need to write a program to get the HTML of a webpage and store it into a string variable.

HoldOffHunger
  • 18,769
  • 10
  • 104
  • 133
kamiar3001
  • 2,646
  • 4
  • 42
  • 78

5 Answers5

124

You can use the WebClient

Using System.Net;

using(WebClient client = new WebClient()) {
    string downloadString = client.DownloadString("http://www.gooogle.com");
}
Fritz
  • 831
  • 7
  • 23
dhinesh
  • 4,676
  • 2
  • 26
  • 44
73

I've run into issues with Webclient.Downloadstring before. If you do, you can try this:

WebRequest request = WebRequest.Create("http://www.google.com");
WebResponse response = request.GetResponse();
Stream data = response.GetResponseStream();
string html = String.Empty;
using (StreamReader sr = new StreamReader(data))
{
    html = sr.ReadToEnd();
}
Scott
  • 13,735
  • 20
  • 94
  • 152
  • 6
    Can you elaborate on the problem you had? – Greg Dec 22 '10 at 14:54
  • 17
    @Greg, it was a performance-related issue. I never really resolved it, but WebClient.DownloadString would take 5-10 seconds to pull down the HTML, where as WebRequest/WebResponse was almost immediate. Just wanted to propose another alternate solution in case the OP had similar issues or wanted a little more control over the request/response. – Scott Dec 22 '10 at 15:00
  • 7
    @Scott - +1 for finding this. Just run some tests. DownloadString took much longer on first use (5299ms downloadstring vs 200ms WebRequest). Tested it in a loop over 50 x BBC, 50 x CNN and 50 x Another RSS feed Urls, using different Urls to avoid caching. After initial load, DownloadString came out 20ms quicker for BBC, 300ms quicker on CNN. For the other RSS feed, WebRequest was 3ms quicker. Generally, I think I'll use WebRequest for singles and DownloadString for looping through URLs. – JsAndDotNet May 02 '13 at 13:28
  • 4
    This worked perfectly for me, thanks! Just to maybe save others a little searching, WebRequest is in System.Net and Stream is in System.Io – Eric Barr Nov 07 '14 at 14:54
  • 1
    Scott, @HockeyJ - I don't know what changed since you used WebClient, but when I tested it (using .NET 4.5.2) it was fast enough - 950ms (still a bit slower than a single WebRequest which took 450 ms but not 5-10 seconds for sure). – BornToCode Sep 14 '16 at 08:26
  • 1
    @BornToCode That's still 2x slower. Not sure I'd double the time it takes to execute a web request just to save a couple of lines of boilerplate code. – Scott Sep 14 '16 at 18:02
  • Take note: as of .Net-7.0: *SYSLIB0014: `WebRequest.Create(string)` is **obsolete**: `WebRequest`, `HttpWebRequest`, `ServicePoint`, and `WebClient` are **obsolete**. Use `HttpClient` instead.'* – NetXpert Sep 17 '22 at 03:24
29

I recommend not using WebClient.DownloadString. This is because (at least in .NET 3.5) DownloadString is not smart enough to use/remove the BOM, should it be present. This can result in the BOM () incorrectly appearing as part of the string when UTF-8 data is returned (at least without a charset) - ick!

Instead, this slight variation will work correctly with BOMs:

string ReadTextFromUrl(string url) {
    // WebClient is still convenient
    // Assume UTF8, but detect BOM - could also honor response charset I suppose
    using (var client = new WebClient())
    using (var stream = client.OpenRead(url))
    using (var textReader = new StreamReader(stream, Encoding.UTF8, true)) {
        return textReader.ReadToEnd();
    }
}
user2246674
  • 7,621
  • 25
  • 28
  • file a bug report – JoelFan Nov 18 '19 at 20:29
  • Take note: as of .Net-7.0: *SYSLIB0014: `WebRequest.Create(string)` is **obsolete**: `WebRequest`, `HttpWebRequest`, `ServicePoint`, and `WebClient` are **obsolete**. Use `HttpClient` instead.'* – NetXpert Sep 17 '22 at 03:26
12
Webclient client = new Webclient();
string content = client.DownloadString(url);

Pass the URL of page who you want to get. You can parse the result using htmlagilitypack.

Laurel
  • 5,965
  • 14
  • 31
  • 57
  • FYI: ```WebClient```, and ```HttpGetRequest``` (amongst others) are **obsolete**. You have to use ```HttpClient``` now, and deal with all of the ```async``` overhead/baggage that it's encumbered with... ‍♀️ – NetXpert Sep 17 '22 at 03:22
5

I have always been using WebClient, but at the time this post is made (.NET 6 is avail), WebClient is getting deprecated.

The preferred way is

HttpClient client = new HttpClient();
string content = await client.GetStringAsync(url);
Ji_in_coding
  • 1,691
  • 1
  • 14
  • 17
  • 3
    @NetXpert I'd suggest running the code block above in a C# Interactive window before declaring it useless or incomplete. I believe the OP is capable of using async code hence the code snippet is not wrapped an async function returning Task. I have updated the 2nd line with variable assignment to make it complete. – Ji_in_coding Sep 18 '22 at 04:16
  • An example of an actually *usable* ```HttpClient``` implementation can be found here: https://stackoverflow.com/questions/1048199/easiest-way-to-read-from-a-url-into-a-string-in-net/73823939#73823939 – NetXpert Oct 08 '22 at 06:50
  • @NetXpert here you go, just those 2 lines of code in action https://youtu.be/iZLJLK0HxgI – Ji_in_coding Oct 09 '22 at 18:01
  • my previous comment can be further generalized to any context where one needs to run async code in a sync function, wrap the async code with Task.Run and wait for the task to complete. I hope this is sufficient to conclude this discussion – Ji_in_coding Oct 09 '22 at 23:47
  • 1
    @NetXpert check the time gap in all posts made by me, it can range anywhere from a day to a year. I have a life, I answer posts on SO when I have nothing better to do. Do not use self-taught as an excuse for not learning your programming fundamentals properly. I am a self taught, so are 99% of all devs out there. You acquire any skill your job demands of you. Now, you have learnt something new, be appreciative and move on. – Ji_in_coding Oct 10 '22 at 02:10