1

I have been trying to perform sentiment analysis over a movie reviews dataset and I am stuck at a point where I am unable to remove english stopwords from the data. What am I doing wrong?

from nltk.corpus import stopwords
stop = stopwords.words("English")
list_ = []
for file_ in dataset:
    dataset['Content'] = dataset['Content'].apply(lambda x: [item for item in x.split(',') if item not in stop])
    list_.append(dataset)
dataset = pd.concat(list_, ignore_index=True)
ykombinator
  • 2,724
  • 7
  • 25
  • 46

4 Answers4

1

I think the code should work with information so far. The assumption I am making is with data has extra space while separated with comma. Below is the test ran: (hope it helps!) 

import pandas as pd
from nltk.corpus import stopwords
import nltk

stop = nltk.corpus.stopwords.words('english')

dataset = pd.DataFrame([{'Content':'i, am, the, computer, machine'}])
dataset = dataset.append({'Content':'i, play, game'}, ignore_index=True)
print(dataset)
list_ = []
for file_ in dataset:
    dataset['Content'] = dataset['Content'].apply(lambda x: [item.strip() for item in x.split(',') if item.strip() not in stop])
    list_.append(dataset)
dataset = pd.concat(list_, ignore_index=True)

print(dataset)

Input with stopwords:

                          Content
0   i, am, the, computer, machine
1                   i, play, game

Output:

                Content
 0  [computer, machine]
 1         [play, game]
niraj
  • 17,498
  • 4
  • 33
  • 48
0

Well through your comment I think that you don't need to loop over dataset. (Maybe dataset contains only the single column named Content)

You can simply do:

 dataset["Content"] = dataset["Content"].str.split(",").apply(lambda x: [item for item in x if item not in stop])
0

You are looping over dataset, but appending the whole frame each time and not using file_ Try:

from nltk.corpus import stopwords
stop = stopwords.words("English")
dataset['Cleaned'] = dataset['Content'].apply(lambda x: ','.join([item for item in x.split(',') if item not in stop]))

That returns a Series containing lists of words, if you want to flatten that to a single list:

flat_list = [item for sublist in list(dataset['Cleaned'].values) for item in sublist]

With a hat tip to Making a flat list out of list of lists in Python

tvashtar
  • 4,055
  • 1
  • 14
  • 12
  • I get a `TypeError: string indices must be integers` for this code as well. `dataset` is type `DataFrame` btw. – ykombinator Jun 26 '17 at 03:00
  • Ah ok, that wasn't clear, and what was the form of result you wanted? A single list of words, or a list per row? – tvashtar Jun 26 '17 at 03:15
  • I updated my answer to give you both options. I'm assuming dataset['Content'] elements contain a comma seperated list of words, if not please give an example dataset – tvashtar Jun 26 '17 at 03:21
  • And to clarify you were getting those errors in both examples because iterating over a dataframe, actually iterates over the columns not the rows. For that you can use iterrows, but in this case you can just use apply as shown, since iterrows returns tuples. You could also iterate over the index of dataset if you really wanted to do something like your code. – tvashtar Jun 26 '17 at 03:25
  • Yes the dataset is a comma separated dataframe consisting of movie reviews. Punctuations have been removed from each row. Expected output : 3 rows have say ~50 words, there are 2, 5, 7 stopwords in those rows. Output should be comma separated dataframe of 48, 45 and 43 words. – ykombinator Jun 26 '17 at 03:35
  • Got it, ok modified the answer above to recombine the list of words to a comma separated list in each row – tvashtar Jun 26 '17 at 11:23
0

Try earthy:

>>> from earthy.wordlist import punctuations, stopwords
>>> from earthy.preprocessing import remove_stopwords
>>> result = dataset['Content'].apply(remove_stopwords)

See https://github.com/alvations/earthy/blob/master/FAQ.md#what-else-can-earthy-do

alvas
  • 115,346
  • 109
  • 446
  • 738