Utf-8 on windows python

Question

I have html file to read parse etc, it's encode on unicode (I saw it with the notepad) but when I tried

infile = open("path", "r") 
infile.read()

it fails and I had the famous error :

UnicodeEncodeError: 'charmap' codec can't encode characters in position xx: character maps to undefined

So for test I tried to copy paste the contain of the file in a new one and save it in utf-8 and then tried to open it with codecs like this :

inFile = codecs.open("path", "r", encoding="utf-8")
outputStream = inFile.read()

But I get this error message :

UnicodeEncodeError : 'charmap' codec can't encode character u'\ufeff' in position 0: charcater maps to undefined

I really don't understand because I was created this file in utf8.

That's a [unicode BOM](https://en.wikipedia.org/wiki/Byte_order_mark#Representations_of_byte_order_marks_by_encoding) it seems to be utf-16, can your try passing `encoding='utf-16'` — EdChum, Sep 21 '15 at 12:17
@EdChum I tried and the response is : > UnicodeError: UTF-16 stream does not start with BOM — taspai, Sep 21 '15 at 12:23
Can you post the raw input data just the first few lines or a link to the file, thanks. Another option is to skip the first couple of characters but really it should be able to open this without issue — EdChum, Sep 21 '15 at 12:45
Are you sure you're getting that error during `.read()`!? The error during read would be "can't **decode**". It sounds like you're getting an error when writing to a file or printing to the terminal — Alastair McCormack, Sep 23 '15 at 07:46
@AlastairMcCormack Yes you're right, I get the error during the print to the terminal ! So all works but the terminal can't print some signs ? — taspai, Sep 23 '15 at 07:59
Correct. Can you update your question so that it reflects the problem? The UnicodeEncodeErrors should contain the line number where the fault is so that you can identify the failing statement — Alastair McCormack, Sep 23 '15 at 08:27

score 6 · Accepted Answer · edited May 23 '17 at 10:27

6

UnicodeEncodeError suggests that the code fails while encoding Unicode text to bytes i.e., your actual code tries to print to Windows console. See Python, Unicode, and the Windows console.

The link above fixes UnicodeEncodeError. The next issue is to find out what character encoding is used by the text in your "path" file. If notepad.exe shows the text correctly then it means that it is either encoded using locale.getprefferedencoding(False) (something like cp1252 on Windows) or the file has BOM.

If you are sure that the encoding is utf-8 then pass it to open() directly. Don't use codecs.open():

with open('path', encoding='utf-8') as file:
    html = file.read()

Sometimes, the input may contain text encoded using multiple (inconsistent) encodings e.g., smart quotes may be encoded using cp1252 while the rest of html is utf-8 -- you could fix it using bs4.UnicodeDammit. See also A good way to get the charset/encoding of an HTTP response in Python

edited May 23 '17 at 10:27

Community

1
1

answered Sep 23 '15 at 19:38

jfs

399,953
195
994
1,670

If Notepad says “Unicode” (as the OP said) it means UTF-16. The other encodings are usually called “ANSI” (cp1252 and friends) and “UTF-8” (which is UTF-8 with BOM). – roeland Sep 24 '15 at 22:05
1

@roeland: yes. *"it's encode on unicode (I saw it with the notepad)"* from the question can be interpreted that way. The issue with that theory is that `codecs.open("path", encoding='utf-8').read()` returns `u'\ufeff'` i.e., `utf-8-sig` is more likely. `'utf-8'` encoding fails for both `BOM_UTF16_BE` and `BOM_UTF16_LE`. – jfs Sep 24 '15 at 22:19
Yeah, the question is a bit confusing as it involves two files, the original file in “Unicode”, and the file he re-saved as “UTF-8”. – roeland Sep 24 '15 at 22:23
@roeland: anyway the issue is `UnicodeEncodeError` i.e., when OP tries to print Unicode text to Windows console. – jfs Sep 24 '15 at 22:26
Aha, I see. That was subtle – roeland Sep 24 '15 at 22:32

score 1 · Answer 2 · answered Sep 21 '15 at 23:33

1

The original file probably uses utf-16 (Windows uses the term UNICODE for that encoding).

UTF-8 encoded files on Windows normally starts with a magic number b"\xef\xbb\xbf" (the UTF-8 encoding of U+FEFF) so applications reading that file know it was saved as UTF-8 and not some ANSI code page. utf8-sig which will automatically discard that character.

answered Sep 21 '15 at 23:33

roeland

5,349
2
14
28

2

As a side-note: Don't use `codecs.open`. On Py3, you can pass an `encoding` argument to regular `open`, and on Py2.7, you can import `io.open` (which is the same as Py3's built-in `open`) and do the same. `codecs.open` has some dumb quirks (e.g. doesn't do universal new line handling). – ShadowRanger Sep 22 '15 at 01:01

score 1 · Answer 3 · edited May 23 '17 at 12:18

In anticipation of the OP to update question to reflect the actual problem, the issue is caused by the encoding of the terminal not being defined.

The Windows console is notoriously poor when it comes to Unicode support. For ultimate support, see https://pypi.python.org/pypi/win_unicode_console. Essentially, install "win_unicode_console" (pip install win_unicode_console). Then at the top of your code:

import win_unicode_console
win_unicode_console.enable()

You may also need to use a suitable font - See https://stackoverflow.com/a/5750227/1554386

As you're using an input with a UTF-8 BOM, you should use the utf_8_sig codec so that the BOM is stripped before working with the contents.

As this is Python 3, you don't need to use the codecs module to set encoding when using open().

Putting it together it would look like:

import win_unicode_console
win_unicode_console.enable()

infile = open("path", "r", encoding="utf_8_sig")

it is best to avoid modifying the script. You could run it using `run` module instead (a part of `win-unicode-console`): `py -m run your-unicode-printing-script.py` or if it is appropriate in your case then put `win_unicode_console.enable()` call into `sitecustomize` or `usercustomize` modules. — jfs, Sep 23 '15 at 19:41

Utf-8 on windows python

3 Answers3

Linked