3

I am trying to extract 1-gram, 2-gram and 3-gram from the train corpus, using RWeka NGramTokenizer function. Unfortunately, getting only 1-grams. There is my code:

train_corpus
# clean-up
cleanset1<- tm_map(train_corpus, tolower)
cleanset2<- tm_map(cleanset1, removeNumbers)
cleanset3<- tm_map(cleanset2, removeWords, stopwords("english"))
cleanset4<- tm_map(cleanset3, removePunctuation)
cleanset5<- tm_map(cleanset4, stemDocument, language="english")
cleanset6<- tm_map(cleanset5, stripWhitespace)

# 1-gram
NgramTokenizer1 <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 1))
train_dtm_tf_1g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer1))
dim(train_dtm_tf_1g)
[1]  5905 15322

# 2-gram
NgramTokenizer2 <- function(x) NGramTokenizer(x, Weka_control(min = 2, max = 2))
train_dtm_tf_2g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer2))
dim(train_dtm_tf_2g)
[1]  5905 15322

# 3-gram
NgramTokenizer3 <- function(x) NGramTokenizer(x, Weka_control(min = 3, max = 3))
train_dtm_tf_3g <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer3))
dim(train_dtm_tf_3g)
[1]  5905 15322

Every time getting the same result, which is obviously, wrong.

# combining together 1-gram, 2-gram and 3-gram from corpus 
    NgramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 3))
train_dtm_tf_ng <- DocumentTermMatrix(cleanset6, control=list(tokenize=NgramTokenizer))
dim(train_dtm_tf_ng)
[1]  5905 15322

# A numeric for the maximal allowed sparsity in the range from bigger zero to smaller one
train_rmspa_m_tf_ng_95<-removeSparseTerms(train_dtm_tf_ng, 0.95)
    [1] 5905  172

# creat bag of words (BOW) vector of these terms for use later
train_BOW_3g_95 <- findFreqTerms(train_rmspa_m_tf_3g_95)

# take a look at the terms that appear in the last 5% of the instances
train_BOW_3g_95

  [1] "avg"        "februari"   "januari"    "level"      "nation"     "per"        "price"     
  [8] "rate"       "report"     "reserv"     "reuter"     "also"       "board"      "export"    
  [15] "march"      "may"        "month"      "oil"        "product"    "total"      "annual"    
  [22] "approv"     "april"      "capit"      "common"     "compani"    "five"       "inc"       
  [29] "increas"    "meet"       "mln"        "record"     "said"       "share"      "sharehold" 
  [36] "stock"      "acquir"     "addit"      "buy"        "chang"      "complet"    "continu" 

     ...

Only 1-grams. I tried to rewrite my comand in the following way:

NgramTokenizer <- function(x) NGramTokenizer(x, Weka_control(min = 1, max = 3))

But did not work out. Also tried to add another line:

options(mc.cores=1)

before NgramTokenizer comand, but no changes. Any help?

QuestSolver
  • 31
  • 2
  • Try this to apply tolower function: cleanset1 <- tm_map(train_corpus, content_transformer(tolower)) – lbcommer Apr 14 '17 at 11:41
  • `NGramTokenizer` works as expected. Make your example minimal and reproducible to get qualified help. – lukeA Apr 14 '17 at 11:57
  • lbcommer, it did not help. Still only 1-grams. – QuestSolver Apr 14 '17 at 12:04
  • lukeA, what do you mean it works as expected? Why dont I have 2-grams and 3-grams? – QuestSolver Apr 14 '17 at 12:04
  • I mean what I said: `as.matrix(TermDocumentMatrix(Corpus(VectorSource(c(txt1="This is my house", txt2="My house is green"))), list(tokenize = function(x) NGramTokenizer(x, Weka_control(min=2, max=2)), tolower=TRUE))) ` gives 2-grams. – lukeA Apr 14 '17 at 12:30
  • lukeA, I executed your code. There is r output: Docs Terms txt1 txt2 house 1 1 this 1 0 green 0 1 Sorry, dont se any 2-grams here – QuestSolver Apr 14 '17 at 12:40
  • my `packageVersion("tm")` is `‘0.6.2’` and my `packageVersion("RWeka")` is `‘0.4.30’`. Using `tm::Terms()` on the output from above yields `"house is" "is green" "is my" "my house" "thi\f\vs is"`, which are 2-grams - as expected. – lukeA Apr 14 '17 at 13:33

1 Answers1

3

I came across the same issue today. It seems "tm_map" is not working well with SimpleCorpus for some reasons.

I changed my code from 

corpus = Corpus(VectorSource(pd_cmnt$QRating_Explaination))

to

corpus = VCorpus(VectorSource(pd_cmnt$QRating_Explaination))

And now it works and gives back 2-gram properly.

legoscia
  • 39,593
  • 22
  • 116
  • 167
Frank Lin
  • 45
  • 4