1

I want all tweets related to a particular an event how can I do it? A rough search suggests it is not possible. There are some paid solutions though. https://gnip.com/sources/twitter/historical/ http://keyhole.co/ Is there any workaround?

Intent: I am doing a project which gets me all tweets related to a particular event. Therefore, I search using some keywords. Though this is not accurate I later on can use some classifier to filter the noise.

Abhishek Bhatia
  • 9,404
  • 26
  • 87
  • 142

2 Answers2

2

Through the public API, it is not possible. However, within Twitter itself, you can search for old Tweets, so it is possible there is a workaround using a click/search bot or web scraping.

I would recommend this post (and the comments) here: https://stackoverflow.com/a/24246840/4131059

The limit on their API (as I myself have done) is 2 years. For their search on their webpage, it is unlimited.

EDIT: It's not code: you can automate harvesting data using search queries via their site, but you cannot use their API to do so. I'll give you how to use the search queries to get what you want:

It's not code: you need manually harvest the data using their search service on their website.

This is how you use it: Keywords: from, since, until

Example: code from:tenderlove since:2010-01-01 until:2012-12-31

Grabs all tweets from the user @tenderlove (a well-known coder) that has the word "code" in the tweet from January 1st, 2010 to December 31st, 2012.

Now if you try this with any API query, it will give you no tweets, because it is outside their date range.

Community
  • 1
  • 1
Alex Huszagh
  • 13,272
  • 3
  • 39
  • 67
  • I don't want to be limited by users. Because in my case I don't know which user posted about say a particular disaster. – Abhishek Bhatia Jun 18 '15 at 06:04
  • Can you provide some initial code, that would be greatly helpful. – Abhishek Bhatia Jun 18 '15 at 06:05
  • I added something: the point is you either have to write code to extract data the website from a given list of search queries, one that probably has a manually clicker to fully expand the list of tweets. In short, it's not a simple question because their API does not support your question. I won't write that code for you. Using their API is simple, but it does not support your goal. So either use the paid service (see if it works first), write your own framework to harvest the data, or find a problem that is solvable (or see if Twitter expands their historical API). – Alex Huszagh Jun 18 '15 at 06:22
  • Thanks so much for advice! Building your and shruti's comments. I used this link https://twitter.com/search-advanced?lang=en to scrape and the got tweets. The only problem I face is that it gives the first n tweets. In browser, I can scroll down, but how to do so in python – Abhishek Bhatia Jun 22 '15 at 13:07
  • You can use a visual clicker or Javascript to do that. Find how more tweets are loaded and just do that until it's done. – Alex Huszagh Jun 23 '15 at 17:11
  • I don't know Javascript, how do you suggest I do this in python using a visual clicker. – Abhishek Bhatia Jun 23 '15 at 18:38
2

You can use a bs4 to parse through your tweets and store the keywords in some file or database

I am doing a similar project and what i have done is cretaed a regex to match the keywords.

Shruti Srivastava
  • 378
  • 1
  • 6
  • 26
  • 1
    Does it parse through all tweets across all users and historically as well. – Abhishek Bhatia Jun 18 '15 at 13:01
  • 1
    you can specify it to do so...what I am doing is parsing it customer by customer depending on the customer that I have passed from SQL table – Shruti Srivastava Jun 18 '15 at 13:11
  • Can you provide some code, that would be greatly helpful. – Abhishek Bhatia Jun 20 '15 at 09:03
  • 1
    "you can specify it to do so" > how? Doing it by user I know a way but not across all twitter due to the underlying limitations. – Abhishek Bhatia Jun 20 '15 at 09:04
  • 1
    As in my code...I can pass all customers from sql table and parse the tweets for each...may b u can share sum code n I can help u out in dat...I cant as its a client project so code sharing isnt possib for me – Shruti Srivastava Jun 20 '15 at 14:12
  • Thanks so much for advice! I used this link `https://twitter.com/search-advanced?lang=en` to scrape and the got tweets. The only problem I face is that it gives the first `n` tweets. In browser, I can scroll down, but how to do so in python. – Abhishek Bhatia Jun 22 '15 at 13:06
  • Code used now: https://github.com/abhigenie92/Tweets_extract/blob/master/get_tweets.py – Abhishek Bhatia Jun 22 '15 at 13:58
  • Please check http://stackoverflow.com/questions/30982176/parse-the-html-code-for-a-whole-webpage-scrolled-down. If possible please help me out with this, I am stuck. – Abhishek Bhatia Jun 25 '15 at 18:10