2

df.review:

de la nada mi ya no se escucha 
I tried to set it up 
It is a good product

The aim is to remove non-English rows. I tried this and this but none work. The below code label all the rows as English mistakenly.

from langdetect import detect
df['detect'] = detect(str(df['review']))

for i in df.detect:
    if i not in ('en'):
        df.drop(i)
SaNa
  • 333
  • 1
  • 3
  • 13

1 Answers1

3

the problem you have is that you are passing the series object to the function which doesn't know how to handle it. to pass the individual values use transform or apply. please see the code below.

df['detect'] = df['review'].apply(detect)

this should give you a column that contains the language of the reviews then if you want to drop the columns you do this

df = df[df['detect'] == 'en']

if you got the exception "No features in the text" it means some rows in the column contain no letters. in this case, you need to remove these rows. there is no function in python to check whether a string contains at least one letter, to go around this, you can change all the strings in the column to lower case and then check if all letters in the strings are lower case. if there are no letters at all it'll return false. check the code

df['detect'] = df['detect'].str.lower()
df = df[df['detect'].str.islower()]
Qdr
  • 703
  • 5
  • 13