1

I got a set of Twitter status updates and im trying to filter all direct messages, senders and receivers of the latter. My dataframe includes columns for senders and text. Using regular expressions I'm trying filter the receivers out of the text column.

this ist what I got, but its returning some strange results

   WD <- getwd()
   if (!is.null(WD)) setwd(WD)

   load("data.R")
   #http://www.unet.univie.ac.at/~a0406222/data.R

   dmtext <- grep("^@[a-z0-9_]{1,15}", tweets$text, perl=T, value=TRUE,ignore.case=TRUE)
   dm.receiver <- gsub("^@([a-z0-9_]{1,15})[ :,].*$", "\\1", dmtext, perl=T,ignore.case=TRUE)
   dm.sender <- as.character(tweets$from_user[grep("^@[a-z0-9_]{1,15}", tweets$text,       perl=T,ignore.case=TRUE,value=FALSE)])


   dm.df <- data.frame(dm.sender,dm.receiver,dmtext)

   dm.df[1:1000,2]

these are some examples of the bad results I get for dm.receiver

    @insultaofuturo Apesar da proibição, jovens insistem em acampar no Aterro na Rio+20\nhttp://t.co/dCfFHUWV
    @mqtodd Bringing the .green Internet to Rio+20 Summit | DotGreen\nhttp://t.co/pQqYilXp #RioPlus20 #gogreen
    @Shyman33 Elinor Ostrom's trailblazing commons research can inspire Rio+20\n http://t.co/m7OTHBtP
    @OccupyRio20 @pnud_es @FBuenAbad @rioplussocial #Futurewewant \nALGO DE ESTO SE HA CUMPLIDO? http://t.co/QDlVwT5z
    @UNDP_CDG#UNDP#Asia-Pacific#Rio+20E-discussion on National&amp;Local Planning for Sustainable Development. Contribute&amp;mail:aprc.rio20@undp.org

why is it that I get results longer than 15 characters using {1,15}?

supersambo
  • 811
  • 1
  • 9
  • 25
  • You'll get more helpful answers if you provide a [reproducible example](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) – hadley Oct 09 '12 at 14:02

2 Answers2

1

It turned out to be a encoding problem. I was not able to solve this issue using regular expressions but the software I used to retrieve the tweets has a column which indicates a user id tweets a adressed to. So I'll use this to do the analysis.

supersambo
  • 811
  • 1
  • 9
  • 25
0

Your grep command matches anything that starts with 1-15 alphanumeric characters. For example:

@blahblahblahblahblahblahblahblahblah

will match because grep looks for the start of the line, looks for the @, finds at least one alpha character and then happily stops, considering this a successful match. grep doesn't care what comes after your pattern in the string, as long as it found something that matches.

In order to get only things under 15 characters, you also have to specify what comes next:

dmtext <- grep("^@[a-z0-9_]{1,15}\\b", ...

This matches 1-15 characters, followed by a word boundary (\b, with an extra backslash for string escaping). Thus, it won't match a word that is 16 or 100 characters long--only something with between 1 and 15 characters.

dan1111
  • 6,576
  • 2
  • 18
  • 29
  • the grep command was actually intended to match the whole text. its just a first step reducing my dataset to direct messages. the problem is that the gsub command is also returing long results even though I'm specifying what comes next (a whitespace). – supersambo Oct 09 '12 at 12:39
  • @supersambo, apologies; I misunderstood the issue. – dan1111 Oct 10 '12 at 12:36