SQLite3 : You must not use 8-bit bytestrings unless you use a text_factory

Question

In order to teach myself python, I have been playing with the whois module, and sqlalchemy

I am writing an app which goes through a list of words, and does a lookup for each words with a .tld added. This is working well, but some of the data returns are making SQLite throw the following error

sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. [SQL: u'INSERT INTO domains (domain, country, city, updated_date, expiration_date, registrar) VALUES (?, ?, ?, ?, ?, ?)'] [parameters: (u'<domain removed>', 'DE', 'Hohenm\xf6lsen', datetime.datetime(2013, 12, 30, 0, 0), datetime.datetime(2016, 1, 17, 0, 0), '<provider removed>')]

It seems to be this string:

'Hohenm\xf6lsen'

In an attempt to fix this, I have used a text factory

engine = create_engine('sqlite:///myDB.db')
engine.connect().connection.text_factory = str

I have also tried using utf-8 in my while loop the reads from a dictionary list:

with codecs.open(dictfile, 'rb', encoding='utf-8') as f:

I have also check the variable type for all the entries, and they are all strings, and not arrays or blobs.

My full code can be seen here:

https://github.com/tripledown/domainfinder/blob/master/domainfinder.py

Answer 1

add this line for assigning text_factory. It converts strings into unicode objects.

conn = GetSqliteConnection(db_path) conn.text_factory = lambda x: unicode(x, 'utf-8', 'ignore')