Using replace() to remove full sentences from text data in Python

Question

I am trying to remove three sentences from paragraphs of text data. I have a pandas dataframe with rows of paragraphs that I want to remove the same three sentences from. For example,

import pandas as pd

df_1 = pd.DataFrame({"text": ["the dog is red. He goes outside and runs.", 
                              "i like dogs because they are fun. i don't like that dogs bark at mailmen", 
                              "dogs bark at mailmen and i think its funny."]})
    
custom_stopwords = ["the dog is red", "i like dogs", "dogs bark at mailmen"]
 
for i in custom_stopwords: 
    df_1['text'] = df_1['text'].str.replace(i, '')

This method is working in this example I have provided, but it does not work on my actual data. The data I have is quite large, but I don't see why that would matter in this case. What is happening is some of my sentences will be removed and others will not. For example, I am unable to remove the word "installation(s)" without blocking out the parentheses with "/".

erip · Accepted Answer · 2022-01-18T14:49:16.827

2

pandas.Series.str.replace has a default keyword argument of regex=True which means it assumes the replacements are regular expressions (like your "installation(s)" could be interpreted). You're trying to replace string literals (or non-regular expressions at the very least). Adding regex=False should work fine:

for i in custom_stopwords: 
    df_1['text'] = df_1['text'].str.replace(i, '', regex=False)

edited Jan 18 '22 at 14:49

answered Jan 18 '22 at 14:43

erip

16,374
11
66
121

1

Thank you, that worked perfectly! I wish I had asked sooner rather than spinning my wheels for...hours! – Lily Tromblay Jan 18 '22 at 16:05

score 1 · Answer 2 · answered Jan 18 '22 at 14:43

1

Use str.replace with the argument regex=False. (s) is interpreted as a regular expression group, in this specific case equal to the character s.

answered Jan 18 '22 at 14:43

timgeb

76,762
20
123
145

Thank you, that worked perfectly! – Lily Tromblay Jan 18 '22 at 16:05

Using replace() to remove full sentences from text data in Python

2 Answers2