5

I have a definition that builds a string composed of UTF-8 encoded characters. The output files are opened using 'w+', "utf-8" arguments.

However, when I try to x.write(string) I get the UnicodeEncodeError: 'ascii' codec can't encode character u'\ufeff' in position 1: ordinal not in range(128)

I assume this is because normally for example you would do `print(u'something'). But I need to use a variable and the quotations in u'_' negate that...

Any suggestions?

EDIT: Actual code here:

source = codecs.open("actionbreak/" + target + '.csv','r', "utf-8")
outTarget = codecs.open("actionbreak/" + newTarget, 'w+', "utf-8")
x = str(actionT(splitList[0], splitList[1]))
outTarget.write(x)

Essentially all this is supposed to be doing is building me a large amount of strings that look similar to this:

[日木曜 Deliverables]= CASE WHEN things = 11 THEN C ELSE 0 END

Razzle Dazzle
  • 481
  • 2
  • 8
  • 20

3 Answers3

5

Are you using codecs.open()? Python 2.7's built-in open() does not support a specific encoding, meaning you have to manually encode non-ascii strings (as others have noted), but codecs.open() does support that and would probably be easier to drop in than manually encoding all the strings.

As you are actually using codecs.open(), going by your added code, and after a bit of looking things up myself, I suggest attempting to open the input and/or output file with encoding "utf-8-sig", which will automatically handle the BOM for UTF-8 (see http://docs.python.org/2/library/codecs.html#encodings-and-unicode, near the bottom of the section) I would think that would only matter for the input file, but if none of those combinations (utf-8-sig/utf-8, utf-8/utf-8-sig, utf-8-sig/utf-8-sig) work, then I believe the most likely situation would be that your input file is encoded in a different Unicode format with BOM, as Python's default UTF-8 codec interprets BOMs as regular characters so the input would not have an issue but output could.

Just noticed this, but... when you use codecs.open(), it expects a Unicode string, not an encoded one; try x = unicode(actionT(splitList[0], splitList[1])).

Your error can also occur when attempting to decode a unicode string (see http://wiki.python.org/moin/UnicodeEncodeError), but I don't think that should be happening unless actionT() or your list-splitting does something to the Unicode strings that causes them to be treated as non-Unicode strings.

JAB
  • 20,783
  • 6
  • 71
  • 80
  • Yes, I am using codecs.open(). Even so, it seems like I am running into this error still. – Razzle Dazzle Jul 10 '13 at 18:23
  • @RazzleDazzle It's a BOM issue, then. I made some additions to my answer, try them out and see what works (if anything does). – JAB Jul 10 '13 at 18:32
  • I feel we are getting closer.... following your suggestion to use the encoding that skips the BOM, I now receive this error after about 3 strings are successfully passed into the output file: `UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-3: ordinal not in range(128)` – Razzle Dazzle Jul 10 '13 at 18:37
  • This appears to be the string messing up execution at runtime: `[テスト]= CASE WHEN T THEN c ELSE 0 END` So it seems the japanese characters are causing an encoding error. The input file's encoding is controllable by me. I have been using a UTF-8 encoded .csv as the input in order to preserve the japanese characters – Razzle Dazzle Jul 10 '13 at 18:47
  • @RazzleDazzle It seems I may have discovered the second part of the issue, see my once-again updated answer. – JAB Jul 10 '13 at 18:58
  • encoding it as a UNICODE string allows all the strings to be processed successfully into the output file-- but UTF-8 specific characters do not seem to be preserved in the process. The japanese string I posted earlier now comes out as: `[テスト]= CASE WHEN T` That being said-- you have definitely led me on the right path. Thanks so much! – Razzle Dazzle Jul 10 '13 at 19:02
  • @RazzleDazzle Probably because you're opening the file with a text editor that doesn't interpret the file as UTF-8 if no BOM is present. `テスト` is the cp1252 interpretation of `\xe3\x83\x86\xe3\x82\xb9\xe3\x83\x88`, which is the UTF-8 encoding of `テスト`. Open the output file with the `utf-8-sig` codec as well. – JAB Jul 10 '13 at 19:03
  • Boom. You nailed it. Thank you!! – Razzle Dazzle Jul 10 '13 at 19:05
5

In python 2.x there are two types of string: byte string and unicode string. First one contains bytes and last one - unicode code points. It is easy to determine, what type of string it is - unicode string starts with u:

# byte string
>>> 'abc'
'abc'

# unicode string:
>>> u'abc абв'
u'abc \u0430\u0431\u0432'

'abc' chars are the same, because the are in ASCII range. \u0430 is a unicode code point, it is out of ASCII range. "Code point" is python internal representation of unicode points, they can't be saved to file. It is needed to encode them to bytes first. Here how encoded unicode string looks like (as it is encoded, it becomes a byte string):

>>> s = u'abc абв'
>>> s.encode('utf8')
'abc \xd0\xb0\xd0\xb1\xd0\xb2'

This encoded string now can be written to file:

>>> s = u'abc абв'
>>> with open('text.txt', 'w+') as f:
...     f.write(s.encode('utf8'))

Now, it is important to remember, what encoding we used when writing to file. Because to be able to read the data, we need to decode the content. Here what data looks like without decoding:

>>> with open('text.txt', 'r') as f:
...     content = f.read()
>>> content
'abc \xd0\xb0\xd0\xb1\xd0\xb2'

You see, we've got encoded bytes, exactly the same as in s.encode('utf8'). To decode it is needed to provide coding name:

>>> content.decode('utf8')
u'abc \u0430\u0431\u0432'

After decode, we've got back our unicode string with unicode code points.

>>> print content.decode('utf8')
abc абв
stalk
  • 11,934
  • 4
  • 36
  • 58
1

xgord is right, but for further edification it's worth noting exactly what \ufeff means. It's known as a BOM or a byte order mark and basically it's a callback to the early days of unicode when people couldn't agree which way they wanted their unicode to go. Now all unicode documents are prefaced with either an \ufeff or an \uffef depending on which order they decide to arrange their bytes in.

If you hit an error on those characters in the first location you can be sure the issue is that you are not trying to decode it as utf-8, and the file is probably still fine.

Slater Victoroff
  • 21,376
  • 21
  • 85
  • 144
  • You're slightly incorrect, the BOM will _always_ be `\ufeff`. The actual encoding of the BOM will differ, but the codepoint is always U+FEFF. If you're reading it as `\uffef`, you've got your endianness flipped. – JAB Jul 10 '13 at 18:20
  • (And if it's `b'\xef\xbb\xbf'` then the encoding is UTF-8, of course.) – JAB Jul 10 '13 at 18:38