5

I'm trying to understand Unicode and all asociated things. I have made an utf-8.txt file which obviously is encoded in utf-8. It has "Hello world!" inside. Heres what I do:

f = open('utf8.txt', mode = 'r', encoding = 'utf8')
f.read()

What I get is: '\ufeffHello world!' where did the prefix came from?

Another try:

f = open('utf8.txt', 'rb')
byte = f.read()

printing byte gives: b'\xef\xbb\xbfHello world!' I assume that prefix came in as hex.

byte.decode('utf8')

above code again gives me: '\ufeffHello world!'

What am I doing wrong? How to retrive text to python from utf-8 file?

Thanks for feedback!

Michał Nowosielski
  • 279
  • 1
  • 3
  • 8
  • 2
    Whatever editor you used to save the file, it added an UTF-8 BOM at the beginning of the file, which is explicitly discouraged. Get a better editor. – Matteo Italia Mar 15 '16 at 20:15
  • 1
    Bear in mind, `"Hello world!"` is UTF-8, ASCII, ISO-8859-1, ISO-8859-15, Windows-1252 etc etc etc. Things only getting interesting after `0x7F` – Alastair McCormack Mar 15 '16 at 22:15

1 Answers1

7

Your utf-8.txt is encoded utf-8-bom which is different from utf-8. For an utf-8-bom file, '\uFEFF' is written into the beginning of the file. Instead of using encoding = 'utf8', try encoding = 'utf-8-sig'

f = open('utf8.txt', mode = 'r', encoding = 'utf-8-sig')
print (f.read())
Yunhe
  • 665
  • 5
  • 10