how to preprocess the csv file of tweets?

Question

I have a csv file of 10K tweets that contain all information such as id, location, tweet text,... For sentiment analysis, first I should preprocess these tweets which I use the following code and My machine is IMAC with Python 3.6:

import csv
import re
import numpy as np
from nltk.tokenize import word_tokenize
from string import punctuation
from string import punctuation 
from nltk.corpus import stopwords
class PreProcessTweets:
    def __init__(self):
        self._stopwords = set(stopwords.words('english') + list(punctuation) + ['AT_USER','URL'])

    def processTweets(self, list_of_tweets):
        processedTweets=[]
        for tweet in list_of_tweets:
            processedTweets.append((self._processTweet(tweet["text"]),tweet["label"]))
        return processedTweets

    def _processTweet(self, tweet):
        tweet = tweet.lower() # convert text to lower-case
        tweet = re.sub('((www\.[^\s]+)|(https?://[^\s]+))', 'URL', tweet) # remove URLs
        tweet = re.sub('@[^\s]+', 'AT_USER', tweet) # remove usernames
        tweet = re.sub(r'#([^\s]+)', r'\1', tweet) # remove the # in #hashtag
        tweet = word_tokenize(tweet) # remove repeated characters (helloooooooo into hello)
        return [word for word in tweet if word not in self._stopwords]
testDataSet= open("Twitter_data.csv")
tweetProcessor = PreProcessTweets()

preprocessedTestSet = tweetProcessor.processTweets(testDataSet)

but I get the following error:

TypeError: string indices must be integers

how can I fix this code to correctly preprocess my csv file?

You just iterating through lines of your file in `processTweets` and use them as dictionary object (which they are not), so you get this error. I guess you wanted to use something like `csv.DictReader`, not just `open`. — Nikscorp, Mar 23 '20 at 05:14
Does this answer your question? [Creating a dictionary from a csv file?](https://stackoverflow.com/questions/6740918/creating-a-dictionary-from-a-csv-file) — Nikscorp, Mar 23 '20 at 05:15

how to preprocess the csv file of tweets?

0 Answers0