UnicodeDecodeError: 'utf8' codec can't decode byte 0x9a in position 12

Question

I'm developing a chatbot with the chatterbot library. The chatbot is in my native language --> Slovene, which has a lot of strange characters (for example: š, č, ž). I'm using python 2.7.

When I try to train the bot, the library has trouble with the characters mentioned above. For example, when I run the following code:

chatBot.set_trainer(ListTrainer)
chatBot.train([
            "Koliko imam še dopusta?",
            "Letos imate še 19 dni dopusta.",
        ])

it throws the following error:

UnicodeDecodeError: 'utf8' codec can't decode byte 0x9a in position 12: invalid start byte

I added the # -*- coding: utf-8 -*- line to the top of my file, I also changed the encoding of all used files via my editor (Sublime text 3) to utf-8, I changed the system default encoding with the following code:

import sys
reload(sys)
sys.setdefaultencoding('utf8')

The strings are of type unicode.

When I try to get a response, with these strange characters, it works, it has no issues with them. For example, running the following code in the same execution as the above training code(when I change 'š' to 's' and 'č' to 'c', in the train strings), throws no errors:

chatBot.set_trainer(ListTrainer)
chatBot.train([
            "Koliko imam se dopusta?",
            "Letos imate se 19 dni dopusta.",
        ])    
chatBot.get_response("Koliko imam še dopusta?")

I can't find a solution to this issue. Any suggestions? Thanks loads in advance. :)

EDIT: I used from __future__ import unicode_literals, to make strings of type unicode. I also checked if they really were unicode with the method type(myString)

I would also like to paste this link.

EDIT 2: @MallikarjunaraoKosuri - s code works, but in my case, I had one more thing inside the chatbot instance intialization, which is the following:

chatBot = ChatBot(
    'Test',
    trainer='chatterbot.trainers.ListTrainer',
    storage_adapter='chatterbot.storage.JsonFileStorageAdapter'
)

This is the cause of my error. The json storage file the chatbot creates, is created in my local encoding and not in utf-8. It seems the default storage (.sqlite3), doesn't have this issue, so for now I'll just avoid the json storage. But I am still interested in finding a solution to this error.

Answer 1

The strings from your example are not of type unicode.

Otherwise Python would not throw the UnicodeDecodeError.
This type of error says that at a certain step of program's execution Python tries to decode byte-string into unicode but for some reason fails.

In your case the reason is that:

decoding is configured by utf-8

your source file is not in utf-8 and almost certainly in cp1252:

import unicodedata

b = '\x9a'

# u = b.decode('utf-8') # UnicodeDecodeError: 'utf8' codec can't decode byte 0x9a 
                        # in position 0: invalid start byte

u = b.decode('cp1252')

print unicodedata.name(u) # LATIN SMALL LETTER S WITH CARON
print u # š

So, the 0x9a byte from your cp1252 source can't be decoded with utf-8.

---

The best solution is to do nothing except convertation your source to utf-8.
With Sublime Text 3 you can easily do it by: File -> Reopen with Encoding -> UTF-8.
But don't forget to Ctrl+C your source code before the convertation beacuse just after that all your š, č, ž chars wil be replaced with ?.

Answer 2

Some of our friends are already suggested good part solutions, However again I would like combine all the solutions into one.

And author @gunthercox suggested some guidelines are described here http://chatterbot.readthedocs.io/en/stable/encoding.html#how-do-i-fix-python-encoding-errors

# -*- coding: utf-8 -*-
from chatterbot import ChatBot

# Create a new chat bot named Test
chatBot = ChatBot(
    'Test',
    trainer='chatterbot.trainers.ListTrainer'
)

chatBot.train([
    "Koliko imam še dopusta?",
    "Letos imate še 19 dni dopusta.",
])

Python Terminal

>>> # -*- coding: utf-8 -*-
... from chatterbot import ChatBot
>>> 
>>> # Create a new chat bot named Test
... chatBot = ChatBot(
...     'Test',
...     trainer='chatterbot.trainers.ListTrainer'
... )
>>> 
>>> chatBot.train([
...     "Koliko imam še dopusta?",
...     "Letos imate še 19 dni dopusta.",
... ])
List Trainer: [####################] 100%
>>>