R: running out of memory, how to loop over rows?

Question

I have a dataframe (myDF) with 700.000+ rows, each row has two columns, id and text. The text has 140 character texts (tweets) and I would like to run a sentiment analysis that I got off the web on them. However, no matter what I try, I have memory problems on a macbook with 4gb ram.

I was thinking that maybe I could loop through rows, e.g. do the first 10, and then the second 10...etc. (I run into problems even with batches of 100) Would this solve the problem? What is the best way to loop in such way?

I am posting my code here:

library(plyr)
library(stringr)

# function score.sentiment
score.sentiment = function(sentences, pos.words, neg.words, .progress='none')
{
   # Parameters
   # sentences: vector of text to score
   # pos.words: vector of words of postive sentiment
   # neg.words: vector of words of negative sentiment
   # .progress: passed to laply() to control of progress bar

   # create simple array of scores with laply
   scores = laply(sentences,
   function(sentence, pos.words, neg.words)
   {

      # split sentence into words with str_split (stringr package)
      word.list = str_split(sentence, "\\s+")
      words = unlist(word.list)

      # compare words to the dictionaries of positive & negative terms
      pos.matches = match(words, pos.words)
      neg.matches = match(words, neg.words)

      # get the position of the matched term or NA
      # we just want a TRUE/FALSE
      pos.matches = !is.na(pos.matches)
      neg.matches = !is.na(neg.matches)

      # final score
    score = sum(pos.matches)- sum(neg.matches)
      return(score)
      }, pos.words, neg.words, .progress=.progress )

   # data frame with scores for each sentence
   scores.df = data.frame(text=sentences, score=scores)
   return(scores.df)
}

# import positive and negative words
pos = readLines("positive_words.txt")
neg = readLines("negative_words.txt")

# apply function score.sentiment


myDF$scores = score.sentiment(myDF$text, pos, neg, .progress='text') 

Answer 1

4 GB sounds like enough memory for 700,000 140-character sentences. A different way to calculate your sentiment scores might be more memory and time efficient and / or easier to break into chunks. Instead of processing each sentence, break the entire group of sentences into words

words <- str_split(sentences, "\\s+")

then determine how many words you have in each sentence, and create a single vector of words

len <- sapply(words, length)
words <- unlist(words, use.names=FALSE)

By re-using the words variable I free up the memory used previously for re-cycling (no need to explicitly call the garbage collector, contrary to the advice in @cryo111 !). You can find whether a word is in pos.words or not, without worrying about NAs, with words %in% pos.words. But we can be a bit clever and calculate the cumulative sum of this logical vector, and then subset the cumulative sum at the last word in each sentence

cumsum(words %in% pos.words)[len]

and calculate the number of words as the derivative of this

pos.match <- diff(c(0, cumsum(words %in% pos.words)[len]))

This is the pos.match portion of your score. So

scores <- diff(c(0, cumsum(words %in% pos.words)[len])) - 
          diff(c(0, cumsum(words %in% neg.words)[len]))

and that's it.

score_sentiment <-
    function(sentences, pos.words, neg.words)
{
    words <- str_split(sentences, "\\s+")
    len <- sapply(words, length)
    words <- unlist(words, use.names=FALSE)
    diff(c(0, cumsum(words %in% pos.words)[len])) - 
      diff(c(0, cumsum(words %in% neg.words)[len]))
}

The intention here is that this processes all your sentences in a single pass

myDF$scores <- score_sentiment(myDF$text, pos, neg)

This avoids for loops which, while not inherently inefficient compared to lapply and friends when implemented correctly as indicated by @joran, are very inefficient compared to vectorized solutions. Probably sentences doesn't get copied here, and returning (just) the score won't waste memory returning information (the sentences) we already know about. The biggest memory will be sentences and words.

If memory is still a problem, then I'd create an index that can be used to split the text into smaller groups, and calculate the score of each

nGroups <- 10 ## i.e., about 70k sentences / group
idx <- seq_along(myDF$text)
grp <- split(idx, cut(idx, nGroups, labels=FALSE))
scorel <- lapply(grp, function(i) score_sentiment(myDF$text[i], pos, neg))
myDF$scores <- unlist(scorel, use.names=FALSE)

making sure first that myDF$text is in fact a character, e.g., myDF$test <- as.character(myDF$test)

Answer 2

I think it's rather hard to give a definitive answer to your problem, but here a few pointers. What helped for me was frequent use of the garbage collector gc() as well as removing objects that are no longer needed from the memory rm(obj_name). You could also consider transferring your data into a database such as MySQL. This is rather easy if you export your dataframe as a csv and use LOAD DATA INFILE ... Then it should be possible to loop through much larger chunks than 100 rows (The RODBC package is a good tool to access SQL databases from R). Another alternative would be to hold your data in an external file and to read the data block-wise but I do not know how this can be done effectively in R. It's also useful to keep an eye on the Resource Monitor (Task manager - Performance - Resource Monitor - Memory).

BTW: As far as I have read, a single twitter message can be 560 Bytes long (max). 700k entries gives roughly 400MB of data. Although this is a rather large amount of data, it should be no problem for 4GB RAM. Do you have some other data in your memory? Do you have other programs running?

Answer 3

If I understand correctly, you want to apply a function to sets of ten lines using a loop. Here's a generic way to do that. I first create a list with sets of ten lines using split. They are not ordered, but it should not matter as you can reorder at the end if you want. You then apply your function in a loop and add the results in an 'out' vector using rbind.

x <-matrix(1:100,ncol=1)
parts.start <-split(1:100,1:10) #creates list: divide in 10 sets of 10 lines

out <-NULL
for (i in 1:length(parts.start)){
res <- x[parts.start[[i]],,drop=FALSE]*2 #your function applied to elements of the list.
out <-rbind(out,res)
}
head(out)

     [,1]
[1,]    2
[2,]   22
[3,]   42
[4,]   62
[5,]   82
[6,]  102