Converting Html utf-8 charset to ISO-8859-1 via C#

Question

I've been struggling to convert a html value of an attribute, without any success.

Here is the the HTML i am trying to convert (sure the charset will not be shown here, but, i see it exactly as you see it).

<a href="https://sistemas.usp.br/jupiterweb/listarGradeCurricular?codcg=12&amp;codcur=12012&amp;codhab=1&amp;tipo=N" target="_blank">Administração – São Paulo – diurno</a>

All right, the VALUE of this htmlnode is "Administração - São Paulo - diurno".

I am using HtmlAgilityPack to parse the HtmlPage for this, and once i reach this node, its innerText value is just like this : AdministraÃ§Ã£o â SÃ£o Paulo â diurno

I am assuming the original charset of the page is UTF-8 because thats what the encoding tag on the html says to me.

How can i convert this weird string to : Administração - São Paulo - Diurno ?

I've tried these threads already : thread one and thread two and nothing solved my issue

EDIT: I am getting the page via a C# WebRequest Get.

EDIT2 : Added HtmlAgilityPack tag

The problem was isolated : WebRequest is messing the Html sometimes.

Is there any other way to set the encoding ? i am trying : _webReq.Encoding = "ISO-8859-1"

Thanks in advance

I believe you should tag it with HtmlAgilityPack, looks like that's where the problem is. Are you retrieving this data from a database? If you put a profiler on it, is the data correctly shown? — Andre Calil, Jul 06 '12 at 14:01
I will Edit to make it clearer, and tag it with HtmlAgilityPack. I am getting the page via webrequest c# — Marcello Grechi Lins, Jul 06 '12 at 14:02
That's important as well. I believe you can set the encoding when getting it via WebRequest, give me a minute — Andre Calil, Jul 06 '12 at 14:04
If you set a breakpoint after getting the response, is the content properly shown? I'm trying to isolate the Request or the HtmlAgilityPack. — Andre Calil, Jul 06 '12 at 14:12
No, the html comes "messed" right after the request, so its the WebRequest. Thanks for that, now we know where to focus — Marcello Grechi Lins, Jul 06 '12 at 14:15

score 2 · Accepted Answer · answered Jul 06 '12 at 14:26

After a small test, you can see that the string is not properly getting Encoded back to its original form.

Sample test:

 var item = "Administração - São Paulo - Diurno";
 Console.WriteLine(item);

 var buffer = Encoding.UTF8.GetBytes(item);
 var item2 = Encoding.Default.GetString(buffer);
 Console.WriteLine(item2);

This prints:

Administraçao - Sao Paulo - Diurno
AdministraA§A£o - SA£o Paulo - Diurno

As you can see, the original string is being converted to bytes using UTF8, but then it is being converted back to a string using Default encoding.

This is wrong.

If WebRequest.GetResponse() is returning the string as the wrong value, then there is a problem with that method. Try setting the TransferEncoding property on the HttpWebRequest to UTF8.

Before you can set the TransferEncoding property, you must first set the SendChunked property to true. Clearing TransferEncoding by setting it to null has no effect on the value of SendChunked. Values assigned to the TransferEncoding property replace any existing contents.

Or you can try to set the Encoding to UTF8 on the StreamReader you open. Can I see your code?

score 0 · Answer 2 · answered Apr 20 '17 at 15:11

i solved with:

    HtmlWeb Webget = new HtmlWeb();
    HtmlDocument doc = new HtmlDocument();
    Webget.AutoDetectEncoding = false;
    Webget.OverrideEncoding = Encoding.UTF8;

    doc_tmp.OptionOutputAsXml = true;
    doc_tmp.OptionReadEncoding = true;
    doc_tmp.OptionFixNestedTags = true;
    doc_tmp.OptionDefaultStreamEncoding = Encoding.UTF8;

            doc_tmp.LoadHtml(tmp.InnerHtml);
            doc_tmp.Save(Console.Out);

Converting Html utf-8 charset to ISO-8859-1 via C#

2 Answers2

Linked