3

I would like to use the removeWords (stopwords("english")) function via: corpus <- tm_map(corpus,removeWords, stopwords("english")) but some words like "not", and other negations I'd like to keep.

Is it possible to use the removeWords, stopwords("english") function BUT exclude certain words in that list if specified?

How could I prevent the removal of "not" for example?

(Secondary) is it possible to set this type of control list to all "negations"?

I'd rather not resort to creating my own custom list with only the words from that stoplist that I'm interested in.

Robert
  • 510
  • 1
  • 5
  • 23

1 Answers1

6

You can create a custom list of stopwords by taking the difference between stopwords("en") and the list of words you want to exclude:

exceptions   <- c("not")
my_stopwords <- setdiff(stopwords("en"), exceptions)

If you need to remove all the negations, you can grep them from the stopwords() list:

exceptions <- grep(pattern = "not|n't", x = stopwords(), value = TRUE)
# [1] "isn't"     "aren't"    "wasn't"    "weren't"   "hasn't"    "haven't"   "hadn't"    "doesn't"   "don't"     "didn't"   
# [11] "won't"     "wouldn't"  "shan't"    "shouldn't" "can't"     "cannot"    "couldn't"  "mustn't"   "not"
my_stopwords <- setdiff(stopwords("en"), exceptions)
Duf59
  • 500
  • 6
  • 16
  • `(stopwords("en")`? So `my_stopwords <- setdiff(stopwords("en"), exceptions)` or `my_stopwords <- setdiff(stopwords("english"), exceptions)`? – Robert Oct 27 '15 at 08:35
  • 1
    "en" or "english" gives the same list. – Duf59 Oct 27 '15 at 08:36
  • Would you happen to know how to keep apostrophes in `removePunctuation` as well? I just realized that I would need to know this function as well since I'm including ('). – Robert Oct 27 '15 at 08:41
  • 1
    Just define your own function: `removePunctuationFix <- function(x) gsub("[^[:alnum:][:space:]']", "", x)` There are different ways to write the regex depending on what punctuation marks you want to keep or not, see a discussion [here](http://stackoverflow.com/questions/8697079/remove-all-punctuation-except-apostrophes-in-r) for example. Defining your own function also allows you to replace punctuation with a white space, thus avoiding to merge adjacent words. – Duf59 Oct 27 '15 at 08:55