How can I replace Unicode characters in Python?

Question

I'm pulling Twitter data via their API and one of the tweets has a special character (the right apostrophe) and I keep getting an error saying that Python can't map or character map the character. I've looked all over the Internet, but I have yet to find a solution for this issue. I just want to replace that character with either an apostrophe that Python will recognize, or an empty string (essentially removing it). I'm using Python 3.3. Any input on how to fix this problem? It may seem simple, but I'm a newbie at Python.

Edit: Here is the function I'm using to try to filter out the unicode characters that throw errors.

@staticmethod
def UnicodeFilter(var):
    temp = var
    temp = temp.replace(chr(2019), "'")
    temp = Functions.ToSQL(temp)
    return temp

Also, when running the program, my error is as follows.

'charmap' codec can't encode character '\u2019' in position 59: character maps to 'undefined'

Edit: Here is a sample of my source code:

import json
import mysql.connector
import unicodedata
from MySQLCL import MySQLCL

class Functions(object):
"""This is a class for Python functions"""

@staticmethod
def Clean(string):
    temp = str(string)
    temp = temp.replace("'", "").replace("(", "").replace(")", "").replace(",", "").strip()
    return temp

@staticmethod
def ParseTweet(string):
    for x in range(0, len(string)):
        tweetid = string[x]["id_str"]
        tweetcreated = string[x]["created_at"]
        tweettext = string[x]["text"]
        tweetsource = string[x]["source"]
        truncated = string[x]["truncated"]
        inreplytostatusid = string[x]["in_reply_to_status_id"]
        inreplytouserid = string[x]["in_reply_to_user_id"]
        inreplytoscreenname = string[x]["in_reply_to_screen_name"]
        geo = string[x]["geo"]
        coordinates = string[x]["coordinates"]
        place = string[x]["place"]
        contributors = string[x]["contributors"]
        isquotestatus = string[x]["is_quote_status"]
        retweetcount = string[x]["retweet_count"]
        favoritecount = string[x]["favorite_count"]
        favorited = string[x]["favorited"]
        retweeted = string[x]["retweeted"]
        possiblysensitive = string[x]["possibly_sensitive"]
        language = string[x]["lang"]

        print(Functions.UnicodeFilter(tweettext))
        #print("INSERT INTO tweet(ExTweetID, TweetText, Truncated, InReplyToStatusID, InReplyToUserID, InReplyToScreenName, IsQuoteStatus, RetweetCount, FavoriteCount, Favorited, Retweeted, Language, TweetDate, TweetSource, PossiblySensitive) VALUES (" + str(tweetid) + ", '" + Functions.UnicodeFilter(tweettext) + "', " + str(truncated) + ", " + Functions.CheckNull(inreplytostatusid) + ", " + Functions.CheckNull(inreplytouserid) + ", '" + Functions.CheckNull(inreplytoscreenname) + "', " + str(isquotestatus) + ", " + str(retweetcount) + ", " + str(favoritecount) + ", " + str(favorited) + ", " + str(retweeted) + ", '" + str(language) + "', '" + Functions.ToSQL(tweetcreated) + "', '" + Functions.ToSQL(tweetsource) + "', " + str(possiblysensitive) + ")")
        #MySQLCL.Set("INSERT INTO tweet(ExTweetID, TweetText, Truncated, InReplyToStatusID, InReplyToUserID, InReplyToScreenName, IsQuoteStatus, RetweetCount, FavoriteCount, Favorited, Retweeted, Language, TweetDate, TweetSource, PossiblySensitive) VALUES (" + str(tweetid) + ", '" + tweettext + "', " + str(truncated) + ", " + Functions.CheckNull(inreplytostatusid) + ", " + Functions.CheckNull(inreplytouserid) + ", '" + Functions.CheckNull(inreplytoscreenname) + "', " + str(isquotestatus) + ", " + str(retweetcount) + ", " + str(favoritecount) + ", " + str(favorited) + ", " + str(retweeted) + ", '" + language + "', '" + tweetcreated + "', '" + str(tweetsource) + "', " + str(possiblysensitive) + ")")

@staticmethod
def ToBool(variable):
    if variable.lower() == 'true':
        return True
    elif variable.lower() == 'false':
        return False

@staticmethod
def CheckNull(var):
    if var == None:
        return ""
    else:
        return var

@staticmethod
def ToSQL(var):
    temp = var
    temp = temp.replace("'", "''")
    return str(temp)

@staticmethod
def UnicodeFilter(var):
    temp = var
    #temp = temp.replace(chr(2019), "'")
    unicodestr = unicode(temp, 'utf-8')
    if unicodestr != temp:
        temp = "'"
    temp = Functions.ToSQL(temp)
    return temp

ekhumoro's response was correct.

Thanks for adding a *little* more info, but without knowing how you got the data or exactly which line is generating the error it's hard to help. — Mark Ransom, Mar 18 '16 at 04:51

score 2 · Answer 1 · edited Apr 07 '22 at 14:58

2

you can Encode your unicode string to convert to type str :

a=u"dataàçççñññ"
type(a)
a.encode('ascii','ignore')

this way it will delete the special characters will return you 'data'.

other way you can use unicodedata

edited Apr 07 '22 at 14:58

Campa

4,267
3
37
42

answered Mar 18 '16 at 05:07

aroy

452
2
10

score 2 · Accepted Answer · answered Mar 19 '16 at 19:44

There seem to be two problems with your program.

Firstly, you are passing the wrong code point to chr(). The hexdecimal code point of the character ’ is 0x2019, but you are passing in the decimal number 2019 (which equates to 0x7e3 in hexadecimal). So you need to do either:

    temp = temp.replace(chr(0x2019), "'") # hexadecimal

or:

    temp = temp.replace(chr(8217), "'") # decimal

in order to replace the character correctly.

Secondly, the reason you are getting the error is because some other part of your program (probably the database backend) is trying to encode unicode strings using some encoding other than UTF-8. It's hard to be more precise about this, because you did not include the full traceback in your question. However, the reference to "charmap" suggests a Windows code page is being used (but not cp1252); or an iso encoding (but not iso8859-1, aka latin1); or possibly KOI8_R.

Anyway, the correct way to deal with this issue is to ensure all parts of your program (and especially the database) use UTF-8. If you do that, you won't have to mess about replacing characters anymore.

Funny, I was thinking about a similar solution last night. Haven't been able to try it yet though. — TheRedCameron, Mar 19 '16 at 20:02
Thanks! I tried your first suggestion and all issues are resolved. — TheRedCameron, Mar 20 '16 at 02:29

score -2 · Answer 3 · answered Mar 18 '16 at 03:24

-2

unicode_string = unicode(some_string, 'utf-8')
if unicode_string != some_string:
    some_string = 'whatever you want it to be'

answered Mar 18 '16 at 03:24

quikst3r

1,783
1
10
15

Do you have to import a package for the function 'unicode()'? – TheRedCameron Mar 18 '16 at 03:36
No you don't. It's built in. Basically if the strings aren't the same, then the string is unicode and you can change it – quikst3r Mar 18 '16 at 03:44
Python does not seem to recognize it. "global name 'unicode' is not defined" that's the error i'm getting. – TheRedCameron Mar 18 '16 at 03:45
What version of python are you using? https://www.dropbox.com/s/7cb1n8pqxfj0bu7/Screenshot%202016-03-18%2014.45.18.png?dl=0 – quikst3r Mar 18 '16 at 18:45
As I said in my original post, in using Python 3.3 – TheRedCameron Mar 18 '16 at 21:43

How can I replace Unicode characters in Python?

3 Answers3

Linked