1

I've a json file with plone objects and there is one field of the objects giving me an error:

UnicodeDecodeError('ascii', '{"id":"aluminio-prata", "nome":"ALUM\xc3\x8dNIO PRATA", "num_demaos":0, "rendimento": 0.0, "unidade":"litros", "url":"", "particular":[], "profissional":[], "unidades":[]},', 36, 37, 'ordinal not in range(128)') (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'NoneType' object has no attribute 'getMethodAliases')

I already know witch field is, is the "title" from title = obj.pretty_title_or_id(), when I remove it from here its ok:

json += '{"id":"' + str(id) + '", "nome":"' + title + '", "num_demaos":' + str(num_demaos) + ', "rendimento": ' + str(rendimento) + ', "unidade":"' + str(unidade) + '", "url":"' + link_produto + '", "particular":' + arr_area_particular + ', "profissional":' + arr_area_profissional + ', "unidades":' + json_qtd + '},

but when I leave it I've got this error.

UnicodeDecodeError('ascii', '{"id":"aluminio-prata", "nome":"ALUM\xc3\x8dNIO PRATA", "num_demaos":0, "rendimento": 0.0, "unidade":"litros", "url":"", "particular":[], "profissional":[], "unidades":[]},', 36, 37, 'ordinal not in range(128)') (Also, the following error occurred while attempting to render the standard error message, please see the event log for full details: 'NoneType' object has no attribute 'getMethodAliases')
SteveM
  • 6,058
  • 1
  • 16
  • 20
ccbernardo
  • 33
  • 6
  • How can we help you if we don't know how you generated this CSV and no source has been shared? Did you checked https://opensourcehacker.com/2013/01/04/exporting-plone-content-as-json/ ? – keul Jun 07 '16 at 15:19

2 Answers2

2

I'm going to assume that the error occurs when you're reading the JSON file.

Internally, Plone uses Python Unicode strings for nearly everything. If you read a string from a file, it will need to be decoded into Unicode before Plone can use it. If you give no instructions otherwise, Python will assume that the string was encoded as ASCII, and will attempt its Unicode conversion on that basis. It would be similar to writing:

unicode("ALUM\xc3\x8dNIO PRATA")

which will produce the same kind of error.

In fact, the string you're using was evidently encoded with the UTF-8 character set. That's evident from the "\xc3", and it also makes sense, because that's the character set Plone uses when it sends data to the outside world.

So, how do you fix this? You have to specify the character set that you wish to use when you convert to Unicode:

"ALUM\xc3\x8dNIO PRATA".decode('UTF8')

This gives you a Python Unicode string with no error.

So, after you've read your JSON file into a string (let's call it mystring), you will need to explicitly decode it by using mystring.decode('UTF8'). unicode(mystring, 'UTF8') is another form of the same operation.

SteveM
  • 6,058
  • 1
  • 16
  • 20
0

As Steve already wrote do title.decode('utf8') An Example illustrate the facts:

>>> u"Ä" == u"\xc4"
True              # the native unicode char and escaped versions are the same

>>> "Ä" == u"\xc4"  
False             # the native unicode char is '\xc3\x84' in latin1

>>> "Ä".decode('utf8') == u"\xc4"
True              # one can decode the string to get unicode

>>> "Ä" == "\xc4"
False             # the native character and the escaped string are
                  # of course not equal ('\xc3\x84' != '\xc4').

I find this Thread very helpfull for Problems and Understanding with Encode/Decode of UTF-8.

Community
  • 1
  • 1
1letter
  • 331
  • 2
  • 8