Storing long string of HTML in SQLite database causes unknown error

Question

I am storing some HTML in an SQLite3 database in Python.

When I go to insert some HTML into my SQL table I get an error that I don't understand what's wrong & more importantly how to fix the issue.

Error string:

Exception General: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

The HTML string I am inserting into the table is pretty long (about 700 characters long).

Any idea whats wrong & how I can fix this?

Answer 1

Looking at the answer to this question, it looks like your issue is that you are attempting to insert HTML with characters in it that do not map to ASCII. If you call unicode(my_problematic_html) you'll probably wind up with a UnicodeEncodingError. In that case you'll want to decode your problematic string representation to unicode by calling:

my_unicoded_html = my_problematic_html.decode("utf-8")

and then writing my_unicoded_html to the database.

You'll want to read Unicode In Python Completely Demystified.

_{* Please note, your HTML may be encoded in some other codec (format? ... charset?) than utf-8. latin-1 is also a good guess if you are on Windows (or if the HTML might be from a Windows machine).}