0

I have a file (sample.txt) with the following text (in utf-8):

ولد خألتي وقف في الشارع يقول الله يلعن إلي بيأخذك بيتفل العافيه ˘̩̩⌣˘̩̩ ي أخي حس تالمم لدرجه دي أنا مزعجه ˘̩̩⌣˘̩̩ حرام تحطيمولد خألتي وقف في الشارع يقول الله يلعن إلي بيأخذك بيتفل العافيه ˘̩̩⌣˘̩̩ ي أخي حس تالمم لدرجه دي أنا مزعجه ˘̩̩⌣˘̩̩ حرام تحطيم

When I do the following:

>>> x = open('sample.txt').read().decode('utf-8')

>>> x
u'\ufeff\u0648\u0644\u062f \u062e\u0623\u0644\u062a\u064a \u0648\u0642\u0641 \u0641\u064a \u0627\u0644\u0634\u0627\u0631\u0639 \u064a\u0642\u0648\u0644 \u0627\u
0644\u0644\u0647 \u064a\u0644\u0639\u0646 \u0625\u0644\u064a \u0628\u064a\u0623\u062e\u0630\u0643 \u0628\u064a\u062a\u0641\u0644 \u0627\u0644\u0639\u0627\u0641\
u064a\u0647  \u02d8\u0329\u0329\u2323\u02d8\u0329\u0329 \u064a \u0623\u062e\u064a \u062d\u0633 \u062a\u0627\u0644\u0645\u0645 \u0644\u062f\u0631\u062c\u0647 \u0
62f\u064a \u0623\u0646\u0627 \u0645\u0632\u0639\u062c\u0647  \u02d8\u0329\u0329\u2323\u02d8\u0329\u0329 \u062d\u0631\u0627\u0645 \u062a\u062d\u0637\u064a\u0645\
ufeff\u0648\u0644\u062f \u062e\u0623\u0644\u062a\u064a \u0648\u0642\u0641 \u0641\u064a \u0627\u0644\u0634\u0627\u0631\u0639 \u064a\u0642\u0648\u0644 \u0627\u064
4\u0644\u0647 \u064a\u0644\u0639\u0646 \u0625\u0644\u064a \u0628\u064a\u0623\u062e\u0630\u0643 \u0628\u064a\u062a\u0641\u0644 \u0627\u0644\u0639\u0627\u0641\u06
4a\u0647  \u02d8\u0329\u0329\u2323\u02d8\u0329\u0329 \u064a \u0623\u062e\u064a \u062d\u0633 \u062a\u0627\u0644\u0645\u0645 \u0644\u062f\u0631\u062c\u0647 \u062f
\u064a \u0623\u0646\u0627 \u0645\u0632\u0639\u062c\u0647  \u02d8\u0329\u0329\u2323\u02d8\u0329\u0329 \u062d\u0631\u0627\u0645 \u062a\u062d\u0637\u064a\u0645\n'

>>> open('sample.txt', 'a').write('\n{0}\n'.format(x.encode('utf-8')))

it writes the very same line in the file. I can open it with a text editor, and both are the same, but...

if I do:

print x.encode('utf-8')

I get the following text instead:

n+++ê+ä+» +«+ú+ä+¬+è +ê+é+ü +ü+è +º+ä+¦+º+¦+¦ +è+é+ê+ä +º+ä+ä+ç +è+ä+¦+å +Ñ+ä+è +¿+è+ú+«+¦+â +¿+è+¬+ü+ä +º+ä+¦+º+ü+è+ç -ÿ¦¬¦¬Gîú-ÿ¦¬¦¬ +è +ú+«+è +¡+¦ +¬+º+ä+à+à +ä+»+¦+¼+ç +»+è +ú+å+º +à+¦+¦+¼+ç -ÿ¦¬¦¬Gîú-ÿ¦¬¦¬ +¡+¦+º+à +¬+¡+++è+àn+++ê+ä+» +«+ú+ä+¬+è +ê+é+ü +ü+è +º+ä+¦+º+¦+¦ +è+é+ê+ä +º+ä+ä+ç +è+ä+¦+å +Ñ+ä+è +¿+è+ú+«+¦+â +¿+è+¬+ü+ä +º+ä+¦+º+ü+è+ç -ÿ¦¬¦¬Gîú-ÿ¦¬¦¬ +è +ú+«+è +¡+¦ +¬+º+ä+à+à +ä+»+¦+¼+ç +»+è +ú+å+º +à+¦+¦+¼+ç -ÿ¦¬¦¬Gîú-ÿ¦¬¦¬ +¡+¦+º+à +¬+¡+++è+à

Any ideas of what is happening?

Oscar Mederos
  • 29,016
  • 22
  • 84
  • 124
  • 1
    Looks like your terminal doesn't support UTF-8. Try redirecting to a file and opening the result in your text editor. – Mat Mar 14 '13 at 19:28
  • Your terminal or console is *not* set to UTF-8 and is interpreting the bytes differently. – Martijn Pieters Mar 14 '13 at 19:29
  • I'm using MS Windows (`cmd.exe`). Is it possible to set it to UTF-8? – Oscar Mederos Mar 14 '13 at 19:30
  • What platform are you on? You could try different encodings like `"utf-16"` or `"utf-32"` … – filmor Mar 14 '13 at 19:30
  • You can change the codepage to UTF-8 by typing `chcp 65001`, but you will still need to set a font that's capable of the characters you want to show. – filmor Mar 14 '13 at 19:34
  • @filmor tried both `utf-16` and `utf-32`, but didn't work. Also, when I do `chcp 65001`, I get an error, even if I print the 'a' character: `LookupError: unknown encoding: cp65001` – Oscar Mederos Mar 14 '13 at 19:41
  • 1
    @OscarMederos: See [Python, Unicode, and the Windows console](http://stackoverflow.com/q/5419) The windows console and UTF-8 is... tricky. – Martijn Pieters Mar 14 '13 at 20:19

1 Answers1

0

Windows cmd.exe encoding is not 'utf-8'. Mine is 'mbcs'.

You can check your windows encoding with this:

>>> import sys
>>> sys.getfilesystemencoding()

So you can print x with this code:

>>> import sys
>>> print x.encode(sys.getfilesystemencoding())
Huo
  • 798
  • 3
  • 12
  • 22