Search Twitter API for tweets. Save and clean them up to a new txt file

Question

I'm trying to get the below done, but keep getting an error. I know that I'm getting the information from Twitter because my monthly usage increasing. Any help would be appreciated.

Search Twitter for 5000 tweets mentioning the hashtag #covid and store them in a file titled tweets.txt. You should clean the tweets and make sure that one tweet is stored per line in the file, i.e. the tweets.txt file should exactly have 5000 lines (this can be done by replacing \n in the tweet text with a “ ”).

This is what I have so far:

import tweepy

API_KEY = “My key will be here.”

client = tweepy.Client(API_KEY)

query = "covid"

response = client.search_recent_tweets(query)


for tweet in tweepy.Paginator(client.search_recent_tweets, query=query, max_results=10).flatten(limit=5000):
   
  save_file = open('tweets.txt', 'w')
  for tweet in tweepy:
    tweet = tweet.strip()
    save_file.write(tweet + ' ')
    
print(tweet)

save_file.close()

BTW: you shoud open file before `for`-loops because opening with mode `w` automatically remove previous content. If you want to use it inside for`-loop then you would have to use mode `a` (append). And if you open inside `for`-loop then you should also `close` inside this `for`-loop but you close it after all `for`-loop — furas, Apr 28 '22 at 12:57
why do you use two `for`-loops ? Why do you first run `for tweet in tweepy.Paginator(...)` and later similar `for tweet in tweepy:`? maybe second loop should be `for item in tweet:` ? — furas, Apr 28 '22 at 12:59
You shouldn't be running `save_file = open('tweets.txt', 'w')` more than once. — Michał, Apr 28 '22 at 13:00
My apologies sir. I'm a complete rookie (first every post). I'll do better next time. — A poor kid, Apr 29 '22 at 13:59

score 0 · Answer 1 · answered Apr 28 '22 at 13:12

You didn't show error message so I can only guess what can make problem.

You shouldn't use for tweet in tweepy: but directy tweet.text

# --- before loop ---

save_file = open('tweets.txt', 'w')

# --- loop ---

for tweet in tweepy.Paginator(client.search_recent_tweets, query=query, max_results=10).flatten(limit=5000):

    item = tweet.text.strip().replace('\n', ' ')

    save_file.write(item + '\n')

    print(item)

# --- after loop ---

save_file.close()

score 0 · Answer 2 · answered Apr 29 '22 at 11:31

I believe this to be an assignment question for a class you might be taking. This is because I have come across an assignment document which describes your question word for word. In this case, I would actually urge you to try to figure out the error on your own as this would help you hone your own skills, here are some useful POVs for the same.

As for the answer, I agree with @furas and would provide you with some additional resources that may help in the future.

Tweepy docs (you can push max_results to 100): https://docs.tweepy.org/en/stable/v2_pagination.html and https://docs.tweepy.org/en/stable/client.html#tweepy.Client.search_recent_tweets
Avoiding RT content (might be useful in the future): How to Exclude retweets and replies in a search api?
Twitter documentation for building queries: https://developer.twitter.com/en/docs/twitter-api/tweets/search/integrate/build-a-query#operators

I managed to got it to work. There was a bug with the tweepy package that I had to install. What I needed help with was the error message I was getting, which I didn't even added (very sorry about). — A poor kid, Apr 29 '22 at 13:56

Search Twitter API for tweets. Save and clean them up to a new txt file

2 Answers2