0

I have a dataframe of Twitter handles. When I loop over the handles using the search_tweets function, the loop stops collecting tweets if one of the Twitter handles does not return any results.

I would like to construct the loop such that if no results are returned, it ignores the handle and moves to the next.

My handle dataframe looks like this:

handles=data.frame(`Twitter Handle`=c("@_CHKD","@AIDHC","@BannerChildrens","@BaptistOnline"))

And the loop looks like this:

# Loop through the twitter handles & store the results as individual dataframes
for(handle in twitter_handles) {
  result <- search_tweets(handle, n = 3500 , include_rts = FALSE,retryonratelimit = TRUE)
  result$`Twitter Handle` <- handle
  result$Source <- "Search"

  df_name <- paste(tolower(substring(handle, 2)),"_search")

  if(exists(df_name)) {
    assign(df_name, unique(rbind(get(df_name), result)))
  } else {
    assign(df_name, result)
  }
}

When I run the loop, it throws the following error after it encounters a handle that returns nothing:

Error in fix.by(by.x, x) : 'by' must specify a uniquely valid column

I have tried to search online for a solution but I've not been successful. Any pointers would be very helpful.

Varun
  • 1,211
  • 1
  • 14
  • 31
  • Does it throw a visible error? Or do you just see it 'stop' in the result? – D.sen Jun 06 '18 at 19:22
  • I've updated the question with the error that it throws when I run the loop – Varun Jun 06 '18 at 19:25
  • And do you get an error if you run the `result <- search_tweets(handle, n = 3500 , include_rts = FALSE,retryonratelimit = TRUE)` by itself with a handle you know returns nothing? If no error, what is the return object? – D.sen Jun 06 '18 at 19:29
  • If I run it by itself with the handle I know returns nothing, the exact same error appears – Varun Jun 06 '18 at 19:39
  • ok so you need a error check clause. Maybe try `result <- try(search_tweets(handle, n = 3500 , include_rts = FALSE,retryonratelimit = TRUE))`. Which should skip any handle that throws an error. – D.sen Jun 06 '18 at 19:42
  • I tried. Still no luck! – Varun Jun 06 '18 at 19:45

1 Answers1

1

So for me, I do not see an error when I search_tweets for a handle with no tweets (i.e "@BannerChildrens"), instead I return an empty data.frame of length 0. By adding an if statement you can exclude all handles with no tweets. The following code returns three dataframes ("@_CHKD","@AIDHC","@BaptistOnline") that are in my global environment, with no errors.

handles=data.frame(`Twitter Handle`=c("@_CHKD","@AIDHC","@BannerChildrens","@BaptistOnline"), stringsAsFactors = FALSE)


for(handle in handles$Twitter.Handle) {

  result <- search_tweets(handle, n = 3500 , include_rts = FALSE,retryonratelimit = TRUE)

  if(length(result) != 0){
    result$`Twitter Handle` <- handle
    result$Source <- "Search"

    df_name <- paste0(tolower(substring(handle, 2)),"_search")

    if(exists(df_name)) {
      assign(df_name, unique(rbind(get(df_name), result)))
    } else {
      assign(df_name, result)
    }
  }
}
D.sen
  • 938
  • 5
  • 14
  • Hello D.sen based on your answer to this question, I seem to be getting a strange new error message. Here's a link to the new question I posted on the issue. Would you mind having a quick look please? Thank you! https://stackoverflow.com/questions/51672443/r-authentication-error-when-looping-search-tweets-function-rtweet-package-over – Varun Aug 08 '18 at 15:33
  • I will look at it. However if my answered provided at the time solved your problem (at the time) it is standard practice to accept that answer to close out the issue. – D.sen Aug 08 '18 at 18:05
  • Apologies I usually do accept as soon as the issue is closed. I'll accept this one as you had solved the issue at the time. – Varun Aug 08 '18 at 18:35