4

I have a downloader program that download pages from internet . the encoding of each page is different , some are in UTF-8 and some are Unicode. For example : a that shows 'a' character ; pages full of this characters .We should convert this encodings to normal text .

I used the UnicodeEncoding class in c# , but they do not help me .

How can i decode this encodings to real characters? Is there a class or method that converting this ?

Thanks .

Mahdi Amrollahi
  • 2,930
  • 5
  • 27
  • 37
  • As an aside: there is nothing like "normal text" without encoding. Highly recommended: http://www.joelonsoftware.com/articles/Unicode.html – Alex Jun 12 '10 at 13:30
  • Dear, Kenny please do your homework and correct you question title - UTF8 is one of several ways of encoding Unicode. Also these is no such thing as normal text - probably you wanted to say ANSI or ASCII. In this case it's not possible without loosing data. – sorin Jun 12 '10 at 15:56

3 Answers3

6

That is html-encoded; try HtmlDecode? (you'll need a reference to System.Web.dll)

Marc Gravell
  • 1,026,079
  • 266
  • 2,566
  • 2,900
5

Text in html pages which are in the form of starting with & and ending with ;, are HTML encoded.

You can decode these by using:

string html = ...; //your html
string decoded = System.Web.HttpUtility.HtmlDecode( html );

Also see Characters in string changed after downloading HTML from the internet for code on how to make sure you download the page in the correct character set.

Community
  • 1
  • 1
Mikael Svenson
  • 39,181
  • 7
  • 73
  • 79
1

You're getting confused between HTML/XML escaping and UTF-8/Unicode.

If the page is valid XML, life will be easier - you can just parse it as any other XML document, and then just get the relevant text nodes... all the XML escaping will be "unescaped" when you get the text.

If it's arbitrary - and possibly invalid - HTML then life is a bit harder. You may well want to normalize it into valid HTML first, then parse it and again ask for the text nodes.

If you can give us a more concrete example, it will be easier to advise you.

The HtmlDecode method suggested in other answers may very well be all you need - but you should definitely try to understand what's going on first. For example, you may well want to only decode certain fragments of the HTML - if you decode the whole document, then you could end up with text which looks it contains like HTML tags, but actually just contained text in the original document.

Jon Skeet
  • 1,421,763
  • 867
  • 9,128
  • 9,194