How to read html source from a HTTPS url

Question

i am trying to read the html source code of a https url in c# with the following code:

 WebClient webClient = new WebClient();
 string htmlString = w.DownloadString("https://www.targetUrl.com");

enter image description here

this doesn't work for me as i get encoded html string. I tried using HtmlAgilityPack but with no help.

HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(htmlString);

What does this mean `this doesn't work for me as i get encoded html string`? — I4V, May 30 '13 at 06:09
means its not work for HTTPS link as https://www.targetUrl.com — kavita verma, May 30 '13 at 06:11
`WebClient.DownloadString` doesn't need to do anything special to download from a https address. What do you mean "encoded"? How do you know it's encoded? What does it look like? — Snixtor, May 30 '13 at 06:15
We do not have crystal ball here. What do you expect? what do you see? What exception do you get? — I4V, May 30 '13 at 06:15

score 3 · Accepted Answer · edited May 23 '17 at 11:48

3

That URL is returning a gzip compressed string. WebClient doesn't support this by default, so you'll want to go down to the underlying HttpWebRequest class instead. Blatant rip-off of the answer by feroze over here - Automatically decompress gzip response via WebClient.DownloadData

class MyWebClient : WebClient
{
    protected override WebRequest GetWebRequest(Uri address)
    {
        HttpWebRequest request = base.GetWebRequest(address) as HttpWebRequest;
        request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
        return request;
    }
}

edited May 23 '17 at 11:48

Community

1
1

answered May 30 '13 at 06:31

Snixtor

4,239
2
31
54

yes its also work for http://example.com url but not work with https://example.com – kavita verma May 30 '13 at 07:21
@kavitaverma then download the page with `WebClient.DownloadData` and decompress it by yourself. – I4V May 30 '13 at 07:34

score 0 · Answer 2 · edited Dec 19 '19 at 12:57

0

ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
WebClient webClient = new WebClient();
string htmlString = webClient.DownloadString(url);

edited Dec 19 '19 at 12:57

Guy Luz

3,372
20
43

answered May 30 '13 at 06:12

Damith

62,401
13
102
153

You should probably add some comment with your code to make it clear how is your answer relevant compared to the already accepted answer. – Corentin Pane Dec 19 '19 at 15:28

How to read html source from a HTTPS url

2 Answers2