HttpWebResponse - Encoding problem

Question

I have a problem with encoding. When I get site's source code I have: enter image description here

I set encoding to UTF8 like this:

StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
            string sourceCode = reader.ReadToEnd();

Thanks for your help!

Are you sure that the data you take from stream have UTF8 encoding? its seems that its not true — Anton Semenov, Apr 22 '11 at 13:08
I'm using firebug to check encoding. Content-Type: text/html; charset=UTF-8 — carck3r, Apr 22 '11 at 14:40
I had to comment this line with Accept-Encoding header. It works. — carck3r, May 03 '11 at 13:25

mtmk · Answer 1 · 2018-04-23T00:35:17.033

6

Try to use the encoding specified:

Encoding encoding;
try
{
    encoding = Encoding.GetEncoding(response.CharacterSet);
}
catch (ArgumentException)
{
    // Cannot determine encoding, use dafault
    encoding = Encoding.UTF8;
}

StreamReader reader = new StreamReader(response.GetResponseStream(), encoding);
string sourceCode = reader.ReadToEnd();

If you are accepting gzip somehow, this may help: (Haven't tried it myself and admittedly it doesn't make much sense since your encoding is not gzip?!)

request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

edited Apr 23 '18 at 00:35

answered Apr 22 '11 at 13:14

mtmk

6,176
27
32

Thanks, but it doesn't work. It reads utf-8 encoding (System.Text.UTF8Encoding), but there're the same two chars... – carck3r Apr 22 '11 at 14:56
3

Maybe a long shot but, are you receiving gzip by any chance? – mtmk Apr 24 '11 at 20:43
FWIW, I used the .AutomaticDecompression and when the response was NOT encoded, it still worked fine. – Bill Sambrone May 21 '14 at 22:36

Hassan Faghihi · Answer 2 · 2022-10-11T10:28:05.180

I had the same issue, I tried changing encoding, from the source to the result, and I got nothing. in the end, I come across a thread that leads me to the following... Take look here... .NET: Is it possible to get HttpWebRequest to automatically decompress gzip'd responses?

you need to use the following code, before retrieving the response from the request.

rqst.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;

since once we use accept-encoding 'gzip' or 'deflate', the data get compressed, and turn into data unreadable by us. so we need to decompress them.

score 2 · Answer 3 · answered Apr 22 '11 at 13:13

2

But the response might not be UTF-8. Have you checked the CharacterSet and the ContentType properties of the response object to make sure you're using the right encoding?

In any event, those two characters look like the code page 437 characters for values 03 and 08. It looks like there's some binary data in your data stream.

I would suggest that for debugging, you use Stream.Read to read the first few bytes from the response into a byte array and then examine the values to see what you're getting.

answered Apr 22 '11 at 13:13

Jim Mischel

131,090
20
188
351

response.CharacterSet = "UTF-8"; response.ContentType = "text/html; charset=UTF-8"; – carck3r Apr 22 '11 at 14:43
When I'm debugging I'm getting these values (256 size array): http://lulzimg.com/i19/2e6ee1.png – carck3r Apr 22 '11 at 14:51
That sure looks like binary data to me, especially at the beginning. What were you expecting to receive? – Jim Mischel Apr 22 '11 at 16:13

score 0 · Answer 4 · answered Oct 09 '13 at 10:39

0

Change this line in your code:

using (StreamReader streamReader = new StreamReader(stream, Encoding.GetEncoding(1251)))

it may help you..

answered Oct 09 '13 at 10:39

sizar surani

69
6

HttpWebResponse - Encoding problem

4 Answers4