2

I just started with tm package in R and cannot seem to overcome an issue. Even though my tokenizer functions seem to work right:

uniTokenizer <- function(x) NGramTokenizer(x, Weka_control(min=1, max=1))
biTokenizer <- function(x) NGramTokenizer(x, Weka_control(min=2, max=2))
triTokenizer <- function(x) NGramTokenizer(x, Weka_control(min=3, max=3))

uniTDM <- TermDocumentMatrix(corpus, control=list(tokenize = uniTokenizer))
biTDM <- TermDocumentMatrix(corpus, control=list(tokenize = biTokenizer))
triTDM <- TermDocumentMatrix(corpus, control=list(tokenize = triTokenizer))

when I try to pull 2-grams from biTDM, only 1-grams come up... 

findFreqTerms(biTDM, 50)

 [1] "after"   "and"     "most"    "the"     "were"    "years"   "love"   
 [8] "you"     "all"     "also"    "been"    "did"     "from"    "get"

at the same, the 2-gram function appears to be in tact:

x <- biTokenizer(corpus)
head(x)

[1] "c in"             "in the"           "the years"       
[4] "years thereafter" "thereafter most"  "most of"
Serg Sergeev
  • 21
  • 1
  • 2
    Including a [minimal reproducible example](https://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) in your question will increase your chances of getting an answer. – Samuel Oct 19 '17 at 01:28

1 Answers1

0

I can only assume what the problem is here: NGramTokenizer needs a VCorpus object rather than a Corpus object. 

library(tm)
library(RWeka)

# some dummy text
text <- c("Lorem ipsum dolor sit amet, consetetur sadipscing elitr",
          "sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat",
          "sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum",
          "Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet")

# create a VCorpus
corpus <- VCorpus(VectorSource(text)) 


biTokenizer <- function(x) NGramTokenizer(x, Weka_control(min=2, max=2))


biTDM <- TermDocumentMatrix(corpus, control=list(tokenize = biTokenizer))

print(biTDM$dimnames$Terms)

 [1] "accusam et"            "aliquyam erat"         "amet consetetur"       "at vero"               "clita kasd"            "consetetur sadipscing" "diam nonumy"           "diam voluptua"         "dolor sit"             "dolore magna"         
[11] "dolores et"            "duo dolores"           "ea rebum"              "eirmod tempor"         "eos et"                "est lorem"             "et accusam"            "et dolore"             "et ea"                 "et justo"             
[21] "gubergren no"          "invidunt ut"           "ipsum dolor"           "justo duo"             "kasd gubergren"        "labore et"             "lorem ipsum"           "magna aliquyam"        "no sea"                "nonumy eirmod"        
[31] "sadipscing elitr"      "sanctus est"           "sea takimata"          "sed diam"              "sit amet"              "stet clita"            "takimata sanctus"      "tempor invidunt"       "ut labore"             "vero eos"             
[41] "voluptua at"
clemens
  • 6,653
  • 2
  • 19
  • 31