Stack Overflow
Stack Overflow

Questions tagged [twarc2]

20 questions
2
votes
1 answer

Why is Python Twarc2 freezing on a large file?

I am trying to run Python Twarc hydrate on a very large file of 2,339,076 records but it keeps freezing. I have tried the script on a smaller data set and it works fine. My question is, does Twarc have a maximum number of rows it can process? If so…
frogger
  • 31
  • 4
2
votes
2 answers

Streaming multiple users in Twitter API v2

I am migrating from Twitter APIv1.1 to v2, and trying to stream tweets of users filtered by their ids. I use twarc.client2.Twarc2 in python and build a list of rules in this form: {'value': 'from:user1 or from:user2 or ...'} I have more than…
Shanazar
  • 54
  • 6
1
vote
2 answers

How can I save some json files generated in a for loop as csv?

Sorry, I am new in coding in Python, I would need to save a json file generated in a for loop as csv for each iteration of the loop. I wrote a code that works fine to generate the first csv file but then it is overwritten and I did not find a…
Camilla Ancona
  • 13
  • 3
1
vote
0 answers

Advice on speeding up Twarc / Twitter API follower query

I have an academic research API for Twitter and have been using the Twarc Python library to scrape tweets. For actual tweet scraping it works really well. However, when scraping the followers of accounts it seems incredibly slow. My understanding is…
gdhp
  • 31
  • 1
1
vote
2 answers

Issue installing twarc2 in conda environment

I'm trying to install twarc2 in my conda environment (from here) : conda install -c brown-data-science twarc The above command runs fine, if I try it again I get All requested packages already installed. However when I try to import the module in…
Erwan
  • 1,385
  • 1
  • 12
  • 22
1
vote
1 answer

How to flatten columns in pandas dataframe with some columns as json?

I try to flatten some columns in my dataframe, but unfurtunately it does not work. What would be the correct way of doing this? created_at tweet_hashtag tweet_cashtag 2022-07-23 [{'start': 16, 'end': 27, 'tag': 'blockchain'}, {'start': 28,…
tiktak
  • 11
  • 1
1
vote
1 answer

Length of timeline() of Twitter API

I am trying to get all tweets from a specific user: def get_all_tweets(user_id, DEBUG): # Your bearer token here t = Twarc2(bearer_token="blah") # Initialize a list to hold all the tweepy Tweets alltweets = [] new_tweets = {} …
TaihouKai
  • 133
  • 11
1
vote
0 answers

How to check if a large list of users are friends with a small dictionary of target users on Twitter using Twitter API Python

So I have a list of around 6k usernames that I need to go through and check to see if they are following anyone from a target list of about 20 names. I have no idea how to do this efficiently since it seems that the rate limit caps me at 15 users…
user17065241
  • 11
  • 1
1
vote
0 answers

What is the most efficient way to get all retweets for a list of tweet ids using Twitter API V2

I have a list of tweet ids (tids.csv) for which I need to collect ALL retweets. Since we can’t directly retrieve retweets of a specific tweet in v2 API, I had to get all retweets of a specific user (users of the tweets in my tid file) and then…
mOna
  • 2,341
  • 9
  • 36
  • 60
0
votes
0 answers

Twarc2 - Large Data Set

I'm using Twarc2 to try and get 500,000 tweets and am wondering what the limitations for getting a large data set using Twarc are. It's been a few hours with nothing changing in the command line, which led me to have a few questions. How long should…
user3251224
  • 3
  • 1
0
votes
1 answer

Data collection from twitter with twarc2 and limitation

I want to collect some data from Twitter with twarc2. I want to collect the data between specific duration and limit the collection to a daily basis. For example, between the 1st of July to the 10th of July, limit the collected tweets to 100 tweets…
code_lover
  • 25
  • 5
0
votes
1 answer

Class.main programming style in Python

Can anyone tell me what is this line of code is executing ? if __name__ == "__main__": TwitterHarvester.main(TwitterHarvester, QUEUE, [SEARCH_ROUTING_KEY, TIMELINE_ROUTING_KEY]) This is a python py file containing a class called…
Adhithya Kiran
  • 19
  • 3
0
votes
0 answers

I keep reaching the rate limit after a just single search call in Twitter API

I'm performing a full archive search for tweets with a keyword that come from a list of users. I loop through search queries for each username and check for the keyword 'republican'. The problem is that it will loop through a decent number of…
user17065241
  • 11
  • 1
0
votes
1 answer

twarc2 conversation with some ID values doesn't return output

Example 1 For this tweet: https://twitter.com/UrbanKaoboy/status/1520080172896055296 using twarc2 conversation works fine: twarc2 conversation 1520080172896055296 Example 2 However, for this…
dharmatech
  • 8,979
  • 8
  • 42
  • 88
0
votes
1 answer

Retrieving full text using twarc2

I am retrieving tweets using twarc2 with search terms in the following way: twarc2 search --archive --start-time "2015-01-01" --end-time "2018-12-31" --limit 25000 "faith OR #faith" results.jsonl But the resultant tweets are truncated after a…
kkgarg
  • 1,246
  • 1
  • 12
  • 28
1
2