1

In addition to question R Text mining - how to change texts in R data frame column into several columns with word frequencies? I am wondering how I can manage to make columns with bigrams frequencies instead of just word frequencies. Again, many thanks in advance!

This is the example data frame (thanks to Tyler Rinker).

      person sex adult                                 state code
1         sam   m     0         Computer is fun. Not too fun.   K1
2        greg   m     0               No it's not, it's dumb.   K2
3     teacher   m     1                    What should we do?   K3
4         sam   m     0                  You liar, it stinks!   K4
5        greg   m     0               I am telling the truth!   K5
6       sally   f     0                How can we be certain?   K6
7        greg   m     0                      There is no way.   K7
8         sam   m     0                       I distrust you.   K8
9       sally   f     0           What are you talking about?   K9
10 researcher   f     1         Shall we move on?  Good then.  K10
11       greg   m     0 I'm hungry.  Let's eat.  You already?  K11

Data set above:

library(qdap); DATA
Community
  • 1
  • 1
rdatasculptor
  • 8,112
  • 14
  • 56
  • 81

1 Answers1

2

The dev version of qdap (should go to CRAN within the next few days) does ngrams. For now you'll need to use the dev version. On the toy data set this is fast but on a larger data set such as qdap's mraja1 data set requires ~5 minutes to complete. You could:

  1. Select the bigrams more wisely (i.e., don't use them all as there's going to be a ton)
  2. Wait the time
  3. Run it in parallel
  4. Figure out another way to do this
  5. Get a faster computer

Here's the code to get the dev version of qdap and run the bigram search:

library(devtools)
install_github("qdap", "trinker")
library(qdap)

## this gets the bigrams
bigrams <- sapply(ngrams(DATA$state)[[c("all_n", "n_2")]], paste, collapse=" ")

## This searches by grouping variable for bigram use
termco(DATA$state, DATA$person, bigrams)


## To get raw values
termco(DATA$state, DATA$person, bigrams)[["raw"]]
Tyler Rinker
  • 108,132
  • 65
  • 322
  • 519
  • Thanks Tyler! I think you made my day, again :-). I will try your code as soon as I can. – rdatasculptor May 18 '13 at 18:06
  • It looks very promising! The ngrams functions does the job. However termco gives this error: Error in gsub(paste0(".*?($|'|", paste(paste0("\\", char.keep), collapse = "|"), : assertion 'tree->num_tags == num_tags' failed in executing regexp: file 'tre-compile.c', line 627 Any ideas about what is going wrong? – rdatasculptor May 18 '13 at 19:21
  • No I don't have your data so no idea. You can try `debug(termco)` and see what's going on. – Tyler Rinker May 18 '13 at 19:22
  • I haven't found the cause of the error in the termco yet, but I am very excited about your answer. It helps me a lot. – rdatasculptor May 18 '13 at 19:59
  • It turns out to be a matter of just too many texts and words :-) – rdatasculptor May 18 '13 at 20:25
  • So I assume feeding it batches of the text helps. Can I ask how much text and how many terms caused this error? – Tyler Rinker May 19 '13 at 00:39
  • I first tried it with 2400 texts with aproximately an avarage of 180 words each. My solution for now is: i feed the ngrams() a lot less texts first, say 100. After that I can feed termco all the texts, say 2000. – rdatasculptor May 19 '13 at 09:50
  • The only 'problem' I encounter with the termco function is that it doesn't always count bigrams as I expected it would. If there are words used more than once in a text, e.g. twice the word 'the' and twice the word 'and', termco will take these as two different kinds of bigrams: 'the and.1' and 'the and.2' with a freqency of 1 each. I need (and expected) only 'the and' with a frequency of 2. Is there a way to change this termco outcome? Do you see what I mean? Many thanks in advance again :-) – rdatasculptor May 21 '13 at 21:26
  • This is not the appropriate place to ask this. Please ask the question with a MWE at qdap's [bug reporting](https://github.com/trinker/qdap/issues?page=1&state=open) – Tyler Rinker May 21 '13 at 22:36