Word frequency per document in R

Question

I have this following sample data frame

   comments                date
1 i want to hear that   2010-11-01
2 lets get started      2008-03-25
3 i want to get started 2007-03-14

I want to get word frequency from all the documents and also, i want to store the document number (1, 2 or 3) in which the word appeared in.
The output should be a matrix that would have words in one column, their frequency in other and the document number in 3rd.

I tried the nomal tm package but it isnt working in my case.

Each row in `comments` is a different document? Or you have more documents? How will your desired output look like? Also, did you try some Googling prior posting here? For example [this](http://stackoverflow.com/questions/18101047/list-of-word-frequencies-using-r)? — David Arenburg, Sep 06 '15 at 12:06
Yup I did some googling but I'm not getting the desired results. The example which you've mentioned here talks about converting the all the 'comment' docs in one corpus. But what I want here is a matrix that would have words in one column, their frequency in other and the document number in 3rd (which I think won't be possible if i take all the documents in a corpus). — Akshay Jangra, Sep 06 '15 at 12:18

score 1 · Answer 1 · answered Sep 06 '15 at 13:12

And using the tm package and tidyr

library(tm)
library(tidyr)

df <- data.frame(id = c(1, 2, 3),
                 comments = c("that is that", "lets get started", "i want to get started"),
                 date = as.Date(c("2010-11-01", "2008-03-25", "2007-03-14")), stringsAsFactors = FALSE)

corpus <- Corpus(VectorSource(df$comments))
dtm <- DocumentTermMatrix(corpus, control=list(wordLengths=c(1, Inf)))

my_data <- data.frame(as.matrix(dtm), id = df$id, date = df$date)

outcome <- gather(my_data, words, freq, -id, -date)
head(outcome)

  id       date words freq
1  1 2010-11-01   get    0
2  2 2008-03-25   get    1
3  3 2007-03-14   get    1
4  1 2010-11-01     i    0
5  2 2008-03-25     i    0
6  3 2007-03-14     i    1

Thanks for the response, can you tell me how do I assign id to the documents. As of now, I'm unable to extract the serial numbers — Akshay Jangra, Sep 07 '15 at 04:43
Depends where the serial numbers are located. if they are in front of the table, but do not have a column header, then they are probably row names; try rownames() to extract them. As for assigning, depends on where you want to assign them. In my piece of code in the place of id = .... — phiver, Sep 07 '15 at 08:31

score 1 · Answer 2 · answered Sep 06 '15 at 17:29

I've been working with data.table plus stringi a bit more recently so I thought I'd throw these solutions up that are similar to the dplyr solution but may give a nice speed boost with larger data sets.

dat <- data.frame(
    comments= c("i want to hear that", "lets get started", "i want to get started"),
    date = as.Date(c("2010-11-01", "2008-03-25", "2007-03-14")), stringsAsFactors = FALSE
)


library(data.table); library(stringi)
setDT(dat)

dat[, list(word = unlist(stri_extract_all_words(comments)))][, 
    list(freq=.N), by = 'word'][order(word),]

##       word freq
## 1:     get    2
## 2:    hear    1
## 3:       i    2
## 4:    lets    1
## 5: started    2
## 6:    that    1
## 7:      to    2
## 8:    want    2


dat[, list(word = unlist(stri_extract_all_words(comments))), by="date"][, 
    list(freq=.N), by = c('date', 'word')][order(date, word),]

##           date    word freq
##  1: 2007-03-14     get    1
##  2: 2007-03-14       i    1
##  3: 2007-03-14 started    1
##  4: 2007-03-14      to    1
##  5: 2007-03-14    want    1
##  6: 2008-03-25     get    1
##  7: 2008-03-25    lets    1
##  8: 2008-03-25 started    1
##  9: 2010-11-01    hear    1
## 10: 2010-11-01       i    1
## 11: 2010-11-01    that    1
## 12: 2010-11-01      to    1
## 13: 2010-11-01    want    1

bramtayl · Accepted Answer · 2015-09-06T13:08:07.490

library(dplyr)
library(tidyr)
library(stringi)

word__date = 
  data_frame(
    comments= c("i want to hear that", "lets get started", "i want to get started"),
    date = c("2010-11-01", "2008-03-25", "2007-03-14") %>% as.Date ) %>%
  mutate(word = comments %>% stri_split_fixed(pattern = " ")) %>%
  unnest(word) %>%
  group_by(word, date) %>%
  summarize(count = n())

word = 
  word__date %>%
  summarize(count = sum(count))

Word frequency per document in R

3 Answers3