Word_tokenize twitter data

Question

I am trying to just get the english words out of the text file for a simple word frequency objective. How can I filter out the other strings in the list?

from nltk.tokenize import word_tokenize
words = word_tokenize(message.replace('\n',' '))

print(words)

giving output like this:

['Amazon', 'b', 'maji_opai', 'am\\xcd\\x9ca\\xcd\\x89zon\\xe2\\x80\\xa6', '\\xcb\\x99\\xea\\x92\\xb3\\xe2\\x80\\x8b\\xcb\\x99', 'Amazon', "b'RT", 'WorkingGIrl', 'For', 'people', 'love', 'REAL', 'paperbacks', 'THE', 'PARIS', 'EFFECT', '10', 'right', 'https', '//', 'https', 'Amazon', "b'RT", 'AbsentiaSeries', 'ABSENTIA', 'IS', 'HERE', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', '\\xf0\\x9f\\x91\\x81', 'US', 'UK', 'Australia', 'Germany', 'Ireland', 'Italy', 'Netherlands', 'go', 'https', 'Amazon', "b'RT", 

Answer 1

There's a hand-crafted tweet tokenizer in nltk:

>>> from nltk.tokenize import TweetTokenizer
>>> tt = TweetTokenizer()
>>> tweet = 'Thanks to the historic TAX CUTS that I signed into law, your paychecks are going way UP, your taxes are going way DOWN, and America is once again OPEN FOR BUSINESS! #FakeNews'
>>> tt.tokenize(tweet)
['Thanks', 'to', 'the', 'historic', 'TAX', 'CUTS', 'that', 'I', 'signed', 'into', 'law', ',', 'your', 'paychecks', 'are', 'going', 'way', 'UP', ',', 'your', 'taxes', 'are', 'going', 'way', 'DOWN', ',', 'and', 'America', 'is', 'once', 'again', 'OPEN', 'FOR', 'BUSINESS', '!', '#FakeNews']

Answer 2

You can use a simple list comprehension if you have a specific list of words you are looking for, which would look like this:

words = word_tokenize(message.replace('\n',' '))
word_list = ['amazon', 'b']
filtered_words = [x for x in words if x in word_list]

If you are using python frequently you should delve into list comprehension, it will come up alot

Explanation of how list comprehension works?

http://www.pythonforbeginners.com/basics/list-comprehensions-in-python