python3.6: can i use (\\u\w\w\w\w) to remove all unicode from tweet json

Question

I was doing some twitter mining, and pulled the json of tweets into python3 via pandas

before processing further, i noticed alot of the data was not consistent/clean or even useful to me (for now) so i used regex to make the string of tweet messages consistent or delete the offending item

below is that:

data['full_text'] = data['full_text'].replace('^@ABC(\\u2019s)*[ ,\n\t]*', '', regex=True)
data['full_text'] = data['full_text'].replace('(\\n)', '', regex=True)
data['full_text'] = data['full_text'].replace('(\\t)', '.', regex=True)
data['full_text'] = data['full_text'].replace('(\\u2018)|(\\u2019)', "'", 
regex=True)
data['full_text'] = data['full_text'].replace('(\\u201c)|(\\u201d)', "\"", regex=True)
data['full_text'] = data['full_text'].replace('(\\n)|(\\t)', '', regex=True)

i.e. - remove twitter handle if used at beginning (including punctuation linked to it) - json should have no issue with apostrophes. Keep everything consistent and replace unicode for left/right apostrophe with single ' -some tweets have backslash for quote, others use unicode. keep consistent and replace unicode with \" -delete all tabs -assume all new lines are new sentences so replace them with a fullstop

as far as I'm aware, this this is all that is really needed. things like ~ are likely to be useless, with no real purpose to them. The tweets will also have emoticons that i dont care about (for now)

the rest of the punctuation and these emoticons follow the format \uXXXX where x is a number or letter

so my last line was planning to be the below:

data['full_text'] = data['full_text'].replace('(\\u\w\w\w\w)', "", regex=True)

given the large number of tweets i have, i cant verify if everything worked correctly, which is why if anyone could give some advice?

From my research i kept seeing people post things like:

([\u2600-\u27BF])|([\uD83C][\uDF00-\uDFFF])|([\uD83D][\uDC00-\uDE4F])|([\uD83D][\uDE80-\uDEFF]) 

but when i try these, i also still see emoticons etc left in the json. So why not just use \u\w\w\w\w ??? (especially when used at the end?)

===================================================================== update:

data['full_text'] = data['full_text'].replace('^@ABC(\\u2019s)*[ ,\n\t]*', '', regex=True)
data['full_text'] = data['full_text'].replace('(\\n)', '', regex=True)
data['full_text'] = data['full_text'].replace('(\\t)', '.', regex=True)
data['full_text'] = data['full_text'].replace('(\\u2018)|(\\u2019)', "'", regex=True) 
data['full_text'] = data['full_text'].replace('(\\u201c)|(\\u201d)', "\"", regex=True)
data['full_text'] = data['full_text'].replace('https:\/\/t.co\/(\w{10})', "", regex=True)
import string
data['full_text'] = data['full_text'].replace('[^{}]'.format(string.printable), '', regex=True)

It works thanks to James, although I'm getting conflicting information. Is the last line appropriate? is it deleting anything more than just unicode?

Answer 1

It looks like you have a misunderstanding of unicode. Unicode is a standard for describing characters/text/emoji/pictoglyphs/etc. That's it. For example,

• the unicode standard for character 0041 (the 65th character since unicode is in hexidecimal) is "the Latin capital letter A".
• the unicode standard for character 2600 is "black sun with rays".

So that's it. Unicode gives a description of what the character should be. It is up to the particular font and encoding to determine if the character is even displayed and what it looks like on screen. For my particular setup (Windows 10, Consolas font in the terminal) Consolas does not have a character that represents '\u2600', so it just displays the default 'missing' character of the confussed Tofu (a box with a question mark in the center).

So how does this relate to your question? The string '\u2600' is not 5 characters but a single character, represented by its unicode hexidecimal code point. That is why a regex of \u\w\w\w\w will not work, because it it looking for 5 characters, but each unicode character is only a single character.

You can test it yourself.

len('\u2600')
# returns
1

If you really want to remove all non-ascii characters, you can just filter out the text you don't want.

import string

df['full_text'] = df['full_text'].replace('[^{}]'.format(string.printable), '', regex=True)