How to convert data downloaded from a web site into the correct encoding?

Question

I've been having issues with data downloaded using a .net WebClient control, in that I seem to have little control over the encoding of the data that I get back from a web server.

The specifics of the question are in the post linked below, but I want to ask the question in a more general sense as the answer is not really helping (not the answerer's fault!).

WebClient.DownloadString() returns string with peculiar characters

The real problem is that supposedly there is no way to detect the encoding of a response from a web server, and the webserver may not respond using the encoding specified in the headers.

If this is true, how do web browsers such as IE, Firefox and Chrome work out how to decode the stream when you use the view source functionality?

It must be possible, this seems like a really fundamental requirement!

Answer 1

the browser probably look for the binary header.

for more info visit http://en.wikipedia.org/wiki/Binary_file

Snippet:

Some binary files contain headers, blocks of metadata used by a computer program to interpret the data in the file. For example, a GIF file can contain multiple images, and headers are used to identify and describe each block of image data. If a binary file does not contain any headers, it may be called a flat binary file.

Answer 2

Not an answer, but there is quite an interesting article about this: http://www.joelonsoftware.com/articles/Unicode.html

There is a chicken and egg situation in that the browser cant determine the encoding until it can read the ContentType element, and it cant read the ContentType element unless it knows the encoding.

I believe IE also has some heuristics for determining the encoding (e.g. if all byte values are below 128, the content might be assumed to be 7bit ascii)

Answer 3

Looks like the same question as:

Detecting the character encoding of an HTTP POST request

You have to check the HTTP headers. If they aren't given to you by the client, then there isn't a whole lot you can do as a server.

Answer 4

i hope this become useful :

first read page as byte array then use StreamReader, because StreamReader internally detect encoding finally use HttpUtility.HtmlDecode to decode it

WebClient client = new WebClient();
byte[] bytes = client.DownloadData(url);

MemoryStream mem = new MemoryStream(bytes);
StreamReader reader = new StreamReader(mem);
string html = reader.ReadToEnd();
html = HttpUtility.HtmlDecode(html);