Save Streaming Twitter Data to CSV

Question

I'm new to Python, and I'm attempting to save data from the streaming Twitter API to a CSV file. I can successfully print content to my console, but I cannot get it to save.

I've done a search on stack and I've found several examples which come very close to answering my question, but none which I've found very adaptable due to my very limited skillset.

My code to print to console is as follows:

import sys
import tweepy

#pass security information to variables
consumer_key=""
consumer_secret=""
access_key = ""
access_secret = ""


#use variables to access twitter
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)

#create an object called 'customStreamListener'

class CustomStreamListener(tweepy.StreamListener):

    def on_status(self, status):
        print status.author.screen_name, status.created_at, status.text


    def on_error(self, status_code):
        print >> sys.stderr, 'Encountered error with status code:', status_code
        return True # Don't kill the stream

    def on_timeout(self):
        print >> sys.stderr, 'Timeout...'
        return True # Don't kill the stream


streamingAPI = tweepy.streaming.Stream(auth, CustomStreamListener())
streamingAPI.filter(track=['russia'])

hd1 · Accepted Answer · 2014-03-24T17:31:06.850

2

This works for me, but let me know if you have trouble in a comment.

import csv
def on_status(self, status): 
    with open('file.txt', 'w') as f: 
        f.write('Author,Date,Text')
        writer = csv.writer(f)
        writer.writerow([status.author.screen_name, status.created_at, status.text])

edited Mar 24 '14 at 17:31

answered Mar 24 '14 at 00:54

hd1

33,938
5
80
91

Thanks for the help! Sadly, your code throws the following error: "AttributeError: '_csv.writer' object has no attribute 'write'". Any idea what the problem might be? – Rucksack Mar 24 '14 at 17:14
Worked just fine. Thanks for the edit. I had some trouble with encoding errors, so I had to slightly modify your code a bit further. Putting my final edit uptop – Rucksack Mar 24 '14 at 21:10
I would love to, but I lack the rep. So sorry! – Rucksack Mar 25 '14 at 00:29
this did not work for me. I ended up copying all the streamed printed text and pasted in the textfile. – Zane Jan 31 '21 at 00:25
@Zane I can help you debug if you're so inclined. – hd1 Jan 31 '21 at 09:05

score 1 · Answer 2 · edited May 23 '17 at 11:46

You'll need to import csv to get it to save to a file if you want to do this (as your question says write to csv.

First you'll have to:

import csv

Then you'll have to open a file to write to and create a writer:

handle=csv.writer(open('file.csv','wb'))

I'd change

def on_status(self, status):
    print status.author.screen_name, status.created_at, status.text

to something like this:

def on_status(self, status):
    print status.author.screen_name, status.created_at, status.text
    handle.writerow(status.author.screen_name, status.created_at, status.text)

Of course you'd have to send the handle to the class or have some method to get the handle from the main to that method.

I'd also check this stack question as well.

You should be careful with the data that is coming in from Twitter, it can have commas in it as well. Looking at this question they explain how to escape any comma that may exist in a string. Or course you'd then have to escape every variable or at least status.text.

NoSQL databases are really the best way to save twitter data IMO. If you were to use something like MongoDB every tweet and accompanying metadata would be a simple JSON object. You wouldn't have to worry about escaping text or the like, just use the API to create a JSON object and insert into the document store. Of course having a NoSQL database is a bit more complex than simply writing to a csv, but it's worth investigating. — WildBill, Mar 24 '14 at 00:48

score 0 · Answer 3 · answered Mar 24 '14 at 00:42

0

When you say "save," is there a specific format you had in mind?

The first solution I can think of would be to just write whatever you print to a text file:

my_file = open("file.txt","w") 
def on_status(self, status):
    a = status.author.screen_name 
    b = status.created_at 
    c = status.text
    my_file.write(a,b,c) 
my_file.close()

I'm somewhat new to Python as well, so this may not be 100% correct, but it's worth a shot.

answered Mar 24 '14 at 00:42

kttr

414
1
3
11

Also, make sure to put the "open file" line of code OUTSIDE the function itself. Otherwise, you'll wipe the file clean of any text it once had every time you run the function. – kttr Mar 24 '14 at 00:43

Save Streaming Twitter Data to CSV

3 Answers3